Synced
add autistic graph (60014ebe4d) fix brave being a bitch on video and news (538a3f6aa9) fix brave being a little shit about pagination (1aa6278853) clarify ports used on 4play tehehe (fdd1b4ab40) systemctl ffffffffuuaaaaaaakkkkk (4de2712567) added 4play 4get config options lol (2d12a9b284) better documentation for 4play (cd31e49785) fix annoying bug with regex matching null like a fucking dumbass (f6a369ed48) fix failstates not triggering properly (e830a8f372)
This commit is contained in:
parent
5929642eae
commit
97698ca324
3 changed files with 141 additions and 68 deletions
|
|
@ -36,7 +36,7 @@ Now, after compiling, you should have a `libcurl-impersonate-ff.so` sitting some
|
|||
sudo systemctl edit php8.4-fpm.service
|
||||
```
|
||||
|
||||
^This will open a text editor. Add the following in there, in between those 2 comments I pasted just for reference:
|
||||
^This will open a text editor. Add the following details in between those 2 comments:
|
||||
|
||||
```sh
|
||||
### Editing /etc/systemd/system/php8.4-fpm.service.d/override.conf
|
||||
|
|
@ -49,13 +49,13 @@ Environment="CURL_IMPERSONATE=firefox117"
|
|||
### Edits below this comment will be discarded
|
||||
```
|
||||
|
||||
Restart php8.4-fpm. (`sudo service php8.4-fpm restart`). To test things out, try making a search on "Yep", they check for SSL. If you get results (or a timeout), that means it works!
|
||||
Restart php8.4-fpm. (`sudo service php8.4-fpm restart`). To test things out, try making a search on "Yep", they check for SSL. If you get results (or a timeout, this piece of shit engine is slow as fuck) that means it works!
|
||||
|
||||
# Robots.txt
|
||||
Make sure you configure this right to optimize your search engine presence! Head over to `/robots.txt` and change the fossq.com domain to your own domain.
|
||||
|
||||
# Server listing
|
||||
To be listed on https://4get.ca/instances , you must contact *any* of the people in the server list and ask them to add you to their list of instances in their configuration. The instance list is distributed, and I don't have control over it.
|
||||
To be listed on https://fossq.com/instances , you must contact *any* of the people in the server list and ask them to add you to their list of instances in their configuration. The instance list is distributed, and I don't have control over it.
|
||||
|
||||
If you see spammy entries in your instances list, simply remove the instance from your list that pushes the offending entries.
|
||||
|
||||
|
|
@ -80,13 +80,90 @@ Done! The scraper you chose should now be using the rotating proxies. When askin
|
|||
## Important!
|
||||
If you ever test out a `socks5` proxy locally on your machine and find out it works but doesn't on your server, try supplying the `socks5_hostname` protocol instead. Hopefully this tip can save you 3 hours of your life!
|
||||
|
||||
# 4play
|
||||
4play is a Firefox extension. I made it, and it allows me to control a Firefox browser through code. More information about the project [here](https://git.lolcat.ca/lolcat/4play). It allows you to scrape Google (& probably more sites in the future)
|
||||
# 4play setup
|
||||
4play is a Firefox extension I wrote that lets you control a browser from 4get. When you do a search, FOSS-Q sends a query to a nodeJS http server that communicates to the browser via websockets. The browser then reports back various payloads used for scraping. It's very similar to tools like puppeteer or playwright, except it uses websockets to communicate with the browser instead of the browser's debugger. This is needed to prevent `navigator.webdriver` from being set to `true` (also avoids a fuckton of other botchecks, that's just the tip of the iceberg).
|
||||
|
||||
1. Install the [firefox extension](https://addons.mozilla.org/en-US/firefox/addon/4play/) onto a clean Firefox profile. Make sure you have a real display connected and that you use real hardware with an user-grade GPU. Websites detect this now.
|
||||
2. Navigate to `/extra/4play/`
|
||||
3. Install the 4play server: `npm install @lawlers/4play`
|
||||
4. Edit the `render-server.js` file. Change the line `var password = "cnc";` so that it uses a secure password.
|
||||
5. Edit `/data/config.php` and update the 4play password. Also update the `FPLAY_EXTERNAL_ENDPOINT` so that it points to the render HTTP server.
|
||||
## Requirements
|
||||
- A dedicated laptop or computer running a full desktop environment (like Windows, Mac, or Linux DEs)
|
||||
- Something to make the computer think it's connected to a screen. You need it, otherwise you'll fallback to software rendering and sites can detect that. You can use:
|
||||
- Your laptop's screen
|
||||
- A turned-on screen connected via HDMI
|
||||
- An EDID adapter to simulate a display (no affiliate links: [HDMI](https://www.aliexpress.com/item/1005011668390064.html), [DisplayPort](https://www.aliexpress.com/item/1005011659561729.html))
|
||||
- Firefox ESR or the latest version of Firefox
|
||||
- The [4play extension](https://addons.mozilla.org/en-US/firefox/addon/4play/) installed
|
||||
|
||||
Now, if you did everything correctly, you should be able to render search pages on scrapers that require it. If your render server sits on a different server, I highly recommend setting up a reverse proxy to add TLS support.
|
||||
## Installation
|
||||
Install the dependencies on the machine you wish to run 4play on. In a sane setup, you usually want 4play to run on a dedicated user-grade machine. For the sake of simplicity, this machine will be referred as the "render server".
|
||||
|
||||
```sh
|
||||
git clone https://git.lolcat.ca/lolcat/4get
|
||||
cd 4get/extra/4play
|
||||
|
||||
# install nodejs 26.x or later if it's not already installed
|
||||
|
||||
npm install @lawlers/4play
|
||||
sudo npm install -g nodemon
|
||||
|
||||
# edit line 7 where it says 'var password = "cnc";'. Use a strong password.
|
||||
nano page-render.js
|
||||
|
||||
# test the server
|
||||
nodemon page-render.js
|
||||
```
|
||||
|
||||
I recommend setting up a systemd service to auto-start the 4play server:
|
||||
```sh
|
||||
sudo nano /etc/systemd/system/fplay.service
|
||||
```
|
||||
|
||||
Paste this in (make sure to update `User` & `WorkingDirectory`)
|
||||
```sh
|
||||
[Unit]
|
||||
Description=4play
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=will
|
||||
WorkingDirectory=/home/will/Desktop/4get/extra/4play/
|
||||
ExecStart=nodemon /home/will/Desktop/4get/extra/4play/page-render.js
|
||||
Restart=always
|
||||
Environment=NODE_ENV=production
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Now, enable the service
|
||||
```sh
|
||||
sudo systemctl enable fplay
|
||||
sudo service fplay start
|
||||
sudo service fplay status # check the status. You should see it running
|
||||
```
|
||||
|
||||
This should expose a webservice located at `http://localhost:3000`. I highly recommend you put this endpoint behind apache2 or nginx to add SSL, if the render server sits on a different network.
|
||||
|
||||
## Setting up Firefox
|
||||
- Install Firefox. You can go for the ESR version or the latest version. You'll figure it out.
|
||||
- Head over to [this page](https://addons.mozilla.org/en-US/firefox/addon/4play/) or search "4play" on the Firefox extension store and install the extension.
|
||||
- Click the settings cog next to the extension name. Enable "Run in Private Windows" and "Allow automatic updates"
|
||||
- Click the extension in the top bar. Modify the `ws://localhost:3030/cnc` URL: replace the `cnc` path with the password you set earlier for the server.
|
||||
- Use a timeout value of `30000` to avoid getting throttled by Firefox during reconnections.
|
||||
|
||||
If done correctly, you should see the red dot turn green in the extension's icon. You're ready to load pages!
|
||||
|
||||
## Setting up 4get
|
||||
Now, you need to tell FOSS-Q to use the 4play service. Edit `data/config.php` and change these:
|
||||
|
||||
```php
|
||||
const FPLAY_PASSWORD = "cnc"; # Change this to your password
|
||||
const FPLAY_EXTERNAL_ENDPOINT = null; # Change null to "https://your-endpoint.com/api/example";, with the quotes and semicolon
|
||||
```
|
||||
|
||||
**IMPORTANT:** By default, `localhost:3030` is what Firefox connects to. The render server is exposed at `localhost:3000`. Don't mix them up, otherwise you will get errors about an "invalid Upgrade header".
|
||||
|
||||
If everything was done correctly, you should be seeing results on the Google scraper! You can set proxies in `data/config.php` for it, just like any other scraper.
|
||||
|
||||
In case it isn't clear, here is a visual representation of what you want to achieve:
|
||||
|
||||
|
||||
|
|
@ -2,9 +2,9 @@ const fplay = require("@lawlers/4play");
|
|||
const http = require("http");
|
||||
|
||||
// 4play settings
|
||||
var port = 3030;
|
||||
var timeout = 30000;
|
||||
var password = "cnc";
|
||||
var port = parseInt(process.env.FOURGET_FPLAY_PASSWORD) || 3030;
|
||||
var timeout = parseInt(process.env.FOURGET_FPLAY_TIMEOUT) || 30000;
|
||||
var password = process.env.FOURGET_FPLAY_PASSWORD || "cnc";
|
||||
|
||||
// ==================================================
|
||||
|
||||
|
|
@ -234,65 +234,20 @@ fplay.event.on("web_response", async function(page){
|
|||
|
||||
var body = page.body.toString();
|
||||
|
||||
var url_match = page.url.match(new RegExp(callback_data.url_match, "i"));
|
||||
var content_match = body.match(new RegExp(callback_data.content_match, "i"));
|
||||
var fail_url_match = page.url.match(new RegExp(callback_data.fail_url_match, "i"));
|
||||
var fail_content_match = body.match(new RegExp(callback_data.fail_content_match, "i"));
|
||||
|
||||
// handle positive response
|
||||
if(
|
||||
(
|
||||
callback_data.url_match === null &&
|
||||
callback_data.content_match === null
|
||||
) ||
|
||||
(
|
||||
url_match &&
|
||||
content_match
|
||||
) ||
|
||||
(
|
||||
url_match &&
|
||||
callback_data.content_match === null
|
||||
) ||
|
||||
(
|
||||
callback_data.url_match === null &&
|
||||
content_match
|
||||
)
|
||||
){
|
||||
|
||||
callback_data.http_res.writeHead(
|
||||
200,
|
||||
{
|
||||
"Content-Type": "text/plain;charset=UTF-8",
|
||||
"X-Proxy": callback_data.proxy.raw,
|
||||
"X-Container-ID": typeof callback_data.container.id == "undefined" ? callback_data.container : callback_data.container.id
|
||||
}
|
||||
);
|
||||
|
||||
callback_data.http_res.end(body);
|
||||
|
||||
callback_q.delete(page.id); // cleanup callback only
|
||||
return;
|
||||
}
|
||||
var url_match = callback_data.url_match !== null ? page.url.match(new RegExp(callback_data.url_match, "i")) : null;
|
||||
var content_match = callback_data.content_match !== null ? body.match(new RegExp(callback_data.content_match, "i")) : null;
|
||||
var fail_url_match = callback_data.fail_url_match !== null ? page.url.match(new RegExp(callback_data.fail_url_match, "i")) : null;
|
||||
var fail_content_match = callback_data.fail_content_match !== null ? body.match(new RegExp(callback_data.fail_content_match, "i")) : null;
|
||||
|
||||
// handle negative response
|
||||
if(
|
||||
(
|
||||
callback_data.fail_url_match !== null &&
|
||||
callback_data.fail_url_match !== null ||
|
||||
callback_data.fail_content_match !== null
|
||||
) &&
|
||||
(
|
||||
(
|
||||
fail_url_match &&
|
||||
fail_content_match
|
||||
) ||
|
||||
(
|
||||
fail_url_match &&
|
||||
callback_data.fail_content_match === null
|
||||
) ||
|
||||
(
|
||||
callback_data.fail_url_match === null &&
|
||||
fail_content_match
|
||||
)
|
||||
fail_url_match !== null ||
|
||||
fail_content_match !== null
|
||||
)
|
||||
){
|
||||
|
||||
|
|
@ -311,6 +266,47 @@ fplay.event.on("web_response", async function(page){
|
|||
await fplay.tab_close(global_ws, page.id);
|
||||
await fplay.container_delete(global_ws, callback_data.container);
|
||||
callback_q.delete(page.id);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
// handle positive response
|
||||
if(
|
||||
(
|
||||
callback_data.url_match === null &&
|
||||
callback_data.content_match === null
|
||||
) ||
|
||||
(
|
||||
callback_data.url_match !== null &&
|
||||
callback_data.content_match !== null &&
|
||||
url_match &&
|
||||
content_match
|
||||
) ||
|
||||
(
|
||||
callback_data.url_match === null &&
|
||||
callback_data.content_match !== null &&
|
||||
content_match
|
||||
) ||
|
||||
(
|
||||
callback_data.url_match !== null &&
|
||||
callback_data.content_match === null &&
|
||||
url_match
|
||||
)
|
||||
){
|
||||
|
||||
callback_data.http_res.writeHead(
|
||||
200,
|
||||
{
|
||||
"Content-Type": "text/plain;charset=UTF-8",
|
||||
"X-Proxy": callback_data.proxy.raw,
|
||||
"X-Container-ID": typeof callback_data.container.id == "undefined" ? callback_data.container : callback_data.container.id
|
||||
}
|
||||
);
|
||||
|
||||
callback_data.http_res.end(body);
|
||||
|
||||
callback_q.delete(page.id); // cleanup callback only
|
||||
return;
|
||||
}
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -388,14 +388,14 @@ class brave{
|
|||
*/
|
||||
$nextpage =
|
||||
$this->fuckhtml
|
||||
->getElementById(
|
||||
->getElementsByClassName(
|
||||
"pagination",
|
||||
"div"
|
||||
);
|
||||
|
||||
if($nextpage){
|
||||
if(count($nextpage) !== 0){
|
||||
|
||||
$this->fuckhtml->load($nextpage);
|
||||
$this->fuckhtml->load($nextpage[0]);
|
||||
|
||||
$nextpage =
|
||||
$this->fuckhtml
|
||||
|
|
@ -1773,14 +1773,14 @@ class brave{
|
|||
|
||||
$nextpage =
|
||||
$this->fuckhtml
|
||||
->getElementById(
|
||||
->getElementsByClassName(
|
||||
"pagination",
|
||||
"div"
|
||||
);
|
||||
|
||||
if($nextpage){
|
||||
if(count($nextpage) !== 0){
|
||||
|
||||
$this->fuckhtml->load($nextpage);
|
||||
$this->fuckhtml->load($nextpage[0]);
|
||||
|
||||
$nextpage =
|
||||
$this->fuckhtml
|
||||
|
|
|
|||
Loading…
Reference in a new issue