The BeautifulSoup constructur in gen_nojs needed to explicitly set
features='lxml' to silence a warning from the library.
Also temporarily disabled the site alts test since the results are too
unreliable. This should be moved to a unit test instead.
Repl.it has deprecated the anonymous experience so I have removed the part of the README stating that users can host on Repl.it without an account. -- @spikecodes
The pagination footer on the results page after page 2 has three actions
(beginning, next, previous). The footer filter was updated to remove
items with more than three actions to fix this.
See #131
* Add ability to configure site alts w/ env vars
Site alternatives (i.e. twitter.com -> nitter.net) can now be configured
using environment variables:
WHOOGLE_ALT_TW='nitter.net' # twitter alt
WHOOGLE_ALT_YT='invidio.us' # youtube alt
WHOOGLE_ALT_IG='bibliogram.art/u' # instagram alt
Updated testing to confirm results have been modified.
* Add site alt vars to docker settings and readme
Heroku app instances have been notoriously bad at having the instance
automatically upgraded to https. This adds a step in the before request
decorator to always upgrade heroku apps, since they're always deployed
with the certificate, but never configured to upgrade automatically.
Fixes#153
The invidious instance has been updated to invidious.snopyta.org, since
this instance is more reliable and has more users according to
instances.invidio.us
All site alternative redirects now redirect without the 'www' subdomain,
since most of the alternative sites don't have this subdomain set up.
The resolution for enabling full support for search + suggestions in
Chrome is to remove the "method" tag altogether for any Chrome based
browser. Any inclusion of this tag seems to break the search suggestion
feature, and makes the user add the search engine manually.
Dark mode, country, interface language, and search language configs
can now be set in the search query by appending each option as a
url parameter.
Supported args are: 'dark', 'lang_search', 'lang_interface', and 'ctry'
Ex: /search?q=%s&dark=1&lang_search=lang_en...
These config settings persist across page navigation and switching
result type, but will be reset if the main search bar is used.
See #144
Environment variables should by default be disabled, since they are
optional and need further configuration by the user before enabling.
Readme was updated to reflect this approach, as well as moving the
documentation for the variables a bit lower and properly linking to them
in other areas of the readme.
The body tag of the home page was previously hidden until the page was
finished loading to prevent a flash of unstyled content, but this broke
functionality for users who disallow javascript. This adds in a new
noscript tag to manually enable visibility of the body element, as well
as automatically displaying the config section (since its visibility is
also typically handled by javascript).
* Use relative links instead of absolute
This allows for hosting under a subpath. For example if you want to host
whoogle at example.com/whoogle, it should work better with a reverse proxy.
* Use relative link for opensearch.xml
Occasionally, Firefox will send the search suggestion
string to the server without a mimetype, resulting in the suggestion
only appearing in Flask's `request.data` field. This field is typically
not used for parsing arguments, as the documentation states:
Contains the incoming request data as string in case it came with a
mimetype Flask does not handle.
This fix captures the bytes object sent to the server and parses it into
a normal query to be used in forming suggestions.
The tor service is now started by calling a script which runs tor
according to the current container user. If the user is root, the script
will begin the tor service as normal. Otherwise, it runs tor as the
current user. This primarily is meant to address the issue with Heroku
builds (which don't have a root user) not being able to start tor as a
service.
Also refactored the rc/ dir to misc/ (with a tor/ subdir) since that
makes more sense.
* Add tor and http/socks proxy support
Allows users to enable/disable tor from the config menu, which will
forward all requests through Tor.
Also adds support for setting environment variables for alternative
proxy support. Setting the following variables will forward requests
through the proxy:
- WHOOGLE_PROXY_USER (optional)
- WHOOGLE_PROXY_PASS (optional)
- WHOOGLE_PROXY_TYPE (required)
- Can be "http", "socks4", or "socks5"
- WHOOGLE_PROXY_LOC (required)
- Format: "<ip address>:<port>"
See #30
* Refactor acquire_tor_conn -> acquire_tor_identity
Also updated travis CI to set up tor
* Add check for Tor socket on init, improve Tor error handling
Initializing the app sends a heartbeat request to Tor to check for
availability, and updates the home page config options accordingly. This
heartbeat is sent on every request, to ensure Tor support can be
reconfigured without restarting the entire app.
If Tor support is enabled, and a subsequent request fails, then a new
TorError exception is raised, and the Tor feature is disabled until a
valid connection is restored.
The max attempts has been updated to 10, since 5 seemed a bit too low
for how quickly the attempts go by.
* Change send_tor_signal arg type, update function doc
send_tor_signal now accepts a stem.Signal arg (a bit cleaner tbh). Also
added the doc string for the "disable" attribute in TorError.
* Fix tor identity logic in Request.send
* Update proxy init, change proxyloc var name
Proxy is now only initialized if both type and location are specified,
as neither have a default fallback and both are required. I suppose the
type could fall back to http, but seems safer this way.
Also refactored proxyurl -> proxyloc for the runtime args in order to
match the Dockerfile args.
* Add tor/proxy support for Docker builds, fix opensearch/init
The Dockerfile is now updated to include support for Tor configuration,
with a working torrc file included in the repo.
An issue with opensearch was fixed as well, which was uncovered during
testing and was simple enough to fix here. Likewise, DDG bang gen was
updated to only ever happen if the file didn't exist previously, as
testing with the file being regenerated every time was tedious.
* Add missing "@" for socks proxy requests
The header template was using Google's classes for the "Whoogle" logo,
which meant keeping up with their list of colors used in the logo. The
template was updated to only ever use the Whoogle logo color.
Accordingly, the logo specific styling in filter.py was removed, since
it is no longer needed.
Also removes all links to the shopping tab, as it seems that the
majority of the links to items are Google specific links (usually
google.com/aclk links without any discernible param for determining the
true location for the link). The shopping page should be addressed
separately with unique filtering/formatting. Further tracking of this
task will be followed in #136.
Adds support for ~12K ddg-style !bang-operators -- for example "!gh <query>" to search GitHub, "!w <query>" to search Wikipedia, etc.
Bang operators are loosely supported in the search suggestion API, but should be improved upon eventually to prioritize more popular bangs. At the moment, most bang suggestions are obscure results that likely aren't being used by the vast majority of users. This is simply due to the fact that no intelligent filtering occurs between matching the input text and the results, it's simply a string comparison against the available bang operator keys.
The full list of bang operators is generated on initialization of the app, under a new and separate directory (`app/static/bangs/`).
Authored by: @marvinborner
Co-authored by: @benbusby
Initialization of the app now includes generation of a ddg-bang json
file, which is used for all bang style searches afterwards.
Also added search suggestion handling for bang json lookup. Queries
beginning with "!" now reference the bang json file to pull all keys
that match.
Updated test suite to include basic tests for bang functionality.
Updated gitignore to exclude bang subdir.
The javascript controller has been updated to include a call to focus
the cursor on the search field. This previously had only been seen on
Firefox, and was assumed to be a weird FF-specific bug. Adding in a
timeout to allow elements to finish loading allows the field to be
focused as expected.
Also updated the README to include clarification for IP address
tracking.
Removed the section concerning which parts of the project would need modification, since it's not always fair to expect someone to know that beforehand.
Improves clarity of the meaning behind the "Country" filter -- Google
seemingly uses this value to only return results that are hosted in a
particular country, as evidenced in the search differences highlighted
in #123. It now mentions that the results are filtered by website
hosting location.
Also, now that invidio.us is shut down, the fallback URL (invidiou.site)
is now used instead.
* Implemented new dark theme
Now uses a dedicated css file for all dark theme color changes, rather
than replacing color codes directly.
Color theme is from discussion in #60.
* Minor link color update
* Added instructions for Android Firefox >=79.0.0
Long pressing on the search bar and selecting "Add search engine" no longer works as of Android Firefox 79.0.0
* Update README.md
* Corrected search strings to use backticks
Reconfigured template to only use method parameter if set to search via
POST request (which is the default).
Apparently Chrome/Chromium based browsers don't like non-GET request
searches, and specifying a method caused Chrome to reject the template
altogether.
Arrow key navigation through search suggestions now populates the input
field with text content from the active selection. Navigating "down"
past the end of the suggestions list returns the active cursor to position 0,
while navigating "up" before the list of suggestions restores the
original search query and removes the active highlight from element 0.
Full implementation of social media alt redirects (twitter/youtube/instagram -> nitter/invidious/bibliogram) depending on configuration.
Verbatim search and option to ignore search autocorrect are now supported as well.
Also cleaned up the javascript side of whoogle config so that it now
uses arrays of available fields for parsing config values instead of manually assigning each
one to a variable.
This doesn't include support for Google Maps -> Open Street Maps, that
seems a bit more involved than the social media redirects were, so it
should likely be a separate effort.
Updated to ensure a child span element is available before running a
test to verify the correct time range for the result. Need to come up
with a better way of ensuring uniform results across multiple tests,
since otherwise periodic changes in the returned results can cause tests
to fail.
Adding support to choose separately the language of search and the one for the interface (allowing a default givent by google).
Co-authored-by: Joao <ramos.joao@protonmail.com>
This is a proof of concept! The code works, but uses hardcoded operators
and may be placed in the wrong file/class.
The best-case scenario would be the possibility to use the 13.000+ ddg
operators, but I don't know if that's possible without having to
redirect to duckduckgo first.
* Project refactor (#85)
* Major refactor of requests and session management
- Switches from pycurl to requests library
- Allows for less janky decoding, especially with non-latin character
sets
- Adds session level management of user configs
- Allows for each session to set its own config -- users with blocked cookies fall back to the "default" profile (same usage as before)
- Updates key gen/regen to more aggressively swap out keys after each
request
* Added ability to save/load configs by name
- New PUT method for config allows changing config with specified name
- New methods in js controller to handle loading/saving of configs
* Result formatting and removal of unused elements
- Fixed question section formatting from results page (added appropriate
padding and made questions styled as italic)
- Removed user agent display from main config settings
* Minor change to save config button label (now "Save As...")
* Fixed issue with "de-pickling" of flask session
Having a gitignore-everything ("*") file within a flask session folder seems to cause a
weird bug where the state of the app becomes unusable from continuously
trying to prune files listed in the gitignore (and it can't prune '*').
* Switched to pickling saved configs
* Updated ad/sponsored content filter and conf naming
Configs are now named with a .conf extension to allow for easier manual
cleanup/modification of named config files
Sponsored content now removed by basic string matching of span content
* Version bump to 0.2.0
* Fixed request.send return style
* Moved custom conf files to their own directory
* Refactored whoogle session mgmt
Now allows a fallback "default" session to be used if a user's browser
is blocking cookies
* Reworked pytest client fixture to support new session mgmt
* Added better multilingual support, updated filter
Results page now includes method for switching to "All Languages" from
whichever language is specified as the primary in the config (see #74).
Also removes the non-Whoogle links from the page footer, leaving only
the page navigation controls
Added support for the date range filter on the results page, though I'd
still recommend using the ":past <unit>" query instead.
* Removed no-cache enforcement, minor styling/formatting improvements
* Improving ad filtering for non-English languages
* Added footer to results page
