Used in header templates for navigating back to the home page when
behind a reverse proxy config where the app is running from a subpath of
a domain (i.e. "https://something/whoogle/")
Fixes#403
On app init, short hashes are generated from file checksums to use for
cache busting. These hashes are added into the full file name and used
to symlink to the actual file contents. These symlinks are loaded in the
jinja templates for each page, and can tell the browser to load a new
file if the hash changes.
This is only in place for css and js files, but can be extended in the
future for other file types if needed.
Introduces a new config element and environment variable
(WHOOGLE_CONFIG_THEME) for setting the theme of the app. Rather than
just having either light or dark, this allows a user to have their
instance use their current system light/dark preference to determine the
theme to use.
As a result, the dark mode setting (and WHOOGLE_CONFIG_DARK) have been
deprecated, but will still work as expected until a system theme has
been chosen.
* Add support for Lingva translations in results
Searches that contain the word "translate" and are normal search queries
(i.e. not news/images/video/etc) now create an iframe to a Lingva url to
translate the user's search using their configured search language.
The Lingva url can be configured using the WHOOGLE_ALT_TL env var, or
will fall back to the official Lingva instance url (lingva.ml).
For more info, visit https://github.com/TheDavidDelta/lingva-translate
* Add basic test for lingva results
* Allow user specified lingva instances through csp frame-src
* Fix pep8 issue
Since the interface language defaults to IP geolocation by google, the
default language is now set to english. Still not sure if this is the
best solution, but at least temporarily should clear up some confusion
for users with instances deployed in countries outside of their own.
Also performed some minor cleanup:
- Updated name of strip_blocked_sites to clean_query
- Added clean_query to list of jinja template functions
- Ensured site block list doesn't contain duplicate filters
The new site filter breaks links to Maps results, so filter.py needed
to be updated to handle these links as a unique case. A new method was
introduced to easily remove any "-site:..." filters from the query,
which is now also used to format queries in the header template rather
than manually removing the blocked site list within the template itself.
Bumps version to 0.5.1 for releasing the bugfix
Fixes#329
* Replace hardcoded strings using translation json file
This introduces a new "translations.json" file under app/static/settings
that is loaded on app init and uses the user config value for interface
language to determine the appropriate strings to use in Whoogle-specific
elements of the UI (primarily only on the home page).
* Verify interface lang can be used for localization
Check the configured interface language against the available
localization dict before attempting to use, otherwise fall back to
english.
Also expanded language names in the languages json file.
* Add test for validating translation language keys
Also adds Spanish translation to json (the only non-English language I
can add and reasonably validate on my own).
* Validate all translations against original keyset, update readme
Readme has been updated to include basic contributing guidelines for
both code and translations.
* add view image option
* prevent whoogle links from opening in a new tab.
* remove view image template on mobile requests
* change loop values to be more robust to the number of images
* Update app/templates/imageresults.html
* fix "Basically the .cvifge class needs width: 100%; in order to expand the search input to fit the form width."
* Update app/templates/imageresults.html
* remove hardcoded string from template
* Add view image config var to app.json
* Add view image config var to whoogle.env
Co-authored-by: jacr13 <ramos.joao@protonmail.com>
Co-authored-by: Ben Busby <benbusby@protonmail.com>
The previous method of removing all site filters from the search query
removed the last letter of the search. This only applies the substring
filter if any site filters are present in the query.
Fixes#306
* Block websites in search results via user config
Adds a new config field "Block" to specify a comma separated list of
websites to block in search results. This is applied for all searches.
* Add test for blocking sites from search results
* Document WHOOGLE_CONFIG_BLOCK usage
* Strip '-site:' filters from query in header template
The 'behind the scenes' site filter applied for blocked sites was
appearing in the query field when navigating between search categories
(all -> images -> news, etc). This prevents the filter from appearing in
all except "images", since the image category uses a separate header.
This should eventually be addressed when the image page can begin using
the standard whoogle header, but until then, the filter will still
appear for image searches.
* Add option to disable changing of configuration
Introduces a test to ensure the correct response code is found when
attempting to update the config when disabled, and ensure default config
is unchanged when posting a new config dict.
Attempting to update the config using the API when disabled now returns
a 403 code + redirect.
Co-authored-by: Ben Busby <benbusby@protonmail.com>
Enforces 0 margin for the search input form on the result page, which
removes the weird gap that is seen by default.
Also made minor changes to the border styling. Desktop searches now have
a single bottom border in dark mode rather than an all around border,
and the border around the mobile search result input was removed
entirely.
This was unfortunately a bit more complex than just adding an HTML reset
button, since reset buttons only "reset" input content to its original
value rather than clearing it. This doesn't work for Whoogle's needs,
since inputs on search result pages are auto populated with the search
content as their default value.
A reset button was introduced anyways, but is controlled by a few lines
of javascript to allow completely clearing the search input. The button
will only appear on mobile searches.
At the moment, it isn't particularly pretty, but is functional. It uses
just a plain "x" character and is always visible on mobile search result
pages. This leaves plenty of room for improvement moving forward.
Fixes#291
* Add custom CSS field to config
This allows users to set/customize an instance's theme and appearance to
their liking. The config CSS field is prepopulated with all default CSS
variable values to allow quick editing.
Note that this can be somewhat of a "footgun" if someone updates the
CSS to hide all fields/search/etc. Should probably add some sort of
bandaid "admin" feature for public instances to employ until the whole
cookie/session issue is investigated further.
* Symlink all app static files to test dir
* Refactor app/misc/*.json -> app/static/settings/*.json
The country/language json files are used for user config settings, so
the "misc" name didn't really make sense. Also moved these to the static
folder to make testing easier.
* Fix light theme variables in dark theme css
* Minor style tweaking
Introduces a new content security policy header for responses to all
requests to reduce the possibility of ip leaks to outside connections.
By default blocks all inline scripts, and only allows content loaded
from Whoogle.
Refactors a few small inline scripting cases in the project to their own
individual scripts.
* Adds the ability to redirect reddit.com to libredd.it using the existing
"site alts" config setting.
This adds the WHOOGLE_ALT_RD environment variable for optionally
redirecting reddit links to libreddit
(https://github.com/spikecodes/libreddit).
* Include libreddit in home page site alt note
With javascript disabled, searches could not be submitted on the results
page using the "Enter" key. Adding a hidden submit button to the header
template resolves this issue.
Introduces a new javascript "utils" file, which includes a check for
matching the query against a set of tracking number regexes on page
load. If a match is found, the script prepends a link to the
(presumably) appropriate tracking page.
Referenced in #98
Dark mode, country, interface language, and search language configs
can now be set in the search query by appending each option as a
url parameter.
Supported args are: 'dark', 'lang_search', 'lang_interface', and 'ctry'
Ex: /search?q=%s&dark=1&lang_search=lang_en...
These config settings persist across page navigation and switching
result type, but will be reset if the main search bar is used.
See #144
The body tag of the home page was previously hidden until the page was
finished loading to prevent a flash of unstyled content, but this broke
functionality for users who disallow javascript. This adds in a new
noscript tag to manually enable visibility of the body element, as well
as automatically displaying the config section (since its visibility is
also typically handled by javascript).
* Use relative links instead of absolute
This allows for hosting under a subpath. For example if you want to host
whoogle at example.com/whoogle, it should work better with a reverse proxy.
* Use relative link for opensearch.xml
Occasionally, Firefox will send the search suggestion
string to the server without a mimetype, resulting in the suggestion
only appearing in Flask's `request.data` field. This field is typically
not used for parsing arguments, as the documentation states:
Contains the incoming request data as string in case it came with a
mimetype Flask does not handle.
This fix captures the bytes object sent to the server and parses it into
a normal query to be used in forming suggestions.
* Add tor and http/socks proxy support
Allows users to enable/disable tor from the config menu, which will
forward all requests through Tor.
Also adds support for setting environment variables for alternative
proxy support. Setting the following variables will forward requests
through the proxy:
- WHOOGLE_PROXY_USER (optional)
- WHOOGLE_PROXY_PASS (optional)
- WHOOGLE_PROXY_TYPE (required)
- Can be "http", "socks4", or "socks5"
- WHOOGLE_PROXY_LOC (required)
- Format: "<ip address>:<port>"
See #30
* Refactor acquire_tor_conn -> acquire_tor_identity
Also updated travis CI to set up tor
* Add check for Tor socket on init, improve Tor error handling
Initializing the app sends a heartbeat request to Tor to check for
availability, and updates the home page config options accordingly. This
heartbeat is sent on every request, to ensure Tor support can be
reconfigured without restarting the entire app.
If Tor support is enabled, and a subsequent request fails, then a new
TorError exception is raised, and the Tor feature is disabled until a
valid connection is restored.
The max attempts has been updated to 10, since 5 seemed a bit too low
for how quickly the attempts go by.
* Change send_tor_signal arg type, update function doc
send_tor_signal now accepts a stem.Signal arg (a bit cleaner tbh). Also
added the doc string for the "disable" attribute in TorError.
* Fix tor identity logic in Request.send
* Update proxy init, change proxyloc var name
Proxy is now only initialized if both type and location are specified,
as neither have a default fallback and both are required. I suppose the
type could fall back to http, but seems safer this way.
Also refactored proxyurl -> proxyloc for the runtime args in order to
match the Dockerfile args.
* Add tor/proxy support for Docker builds, fix opensearch/init
The Dockerfile is now updated to include support for Tor configuration,
with a working torrc file included in the repo.
An issue with opensearch was fixed as well, which was uncovered during
testing and was simple enough to fix here. Likewise, DDG bang gen was
updated to only ever happen if the file didn't exist previously, as
testing with the file being regenerated every time was tedious.
* Add missing "@" for socks proxy requests
The header template was using Google's classes for the "Whoogle" logo,
which meant keeping up with their list of colors used in the logo. The
template was updated to only ever use the Whoogle logo color.
Accordingly, the logo specific styling in filter.py was removed, since
it is no longer needed.
Also removes all links to the shopping tab, as it seems that the
majority of the links to items are Google specific links (usually
google.com/aclk links without any discernible param for determining the
true location for the link). The shopping page should be addressed
separately with unique filtering/formatting. Further tracking of this
task will be followed in #136.
Improves clarity of the meaning behind the "Country" filter -- Google
seemingly uses this value to only return results that are hosted in a
particular country, as evidenced in the search differences highlighted
in #123. It now mentions that the results are filtered by website
hosting location.
Also, now that invidio.us is shut down, the fallback URL (invidiou.site)
is now used instead.
* Implemented new dark theme
Now uses a dedicated css file for all dark theme color changes, rather
than replacing color codes directly.
Color theme is from discussion in #60.
* Minor link color update
Reconfigured template to only use method parameter if set to search via
POST request (which is the default).
Apparently Chrome/Chromium based browsers don't like non-GET request
searches, and specifying a method caused Chrome to reject the template
altogether.
Full implementation of social media alt redirects (twitter/youtube/instagram -> nitter/invidious/bibliogram) depending on configuration.
Verbatim search and option to ignore search autocorrect are now supported as well.
Also cleaned up the javascript side of whoogle config so that it now
uses arrays of available fields for parsing config values instead of manually assigning each
one to a variable.
This doesn't include support for Google Maps -> Open Street Maps, that
seems a bit more involved than the social media redirects were, so it
should likely be a separate effort.
Adding support to choose separately the language of search and the one for the interface (allowing a default givent by google).
Co-authored-by: Joao <ramos.joao@protonmail.com>
* Project refactor (#85)
* Major refactor of requests and session management
- Switches from pycurl to requests library
- Allows for less janky decoding, especially with non-latin character
sets
- Adds session level management of user configs
- Allows for each session to set its own config -- users with blocked cookies fall back to the "default" profile (same usage as before)
- Updates key gen/regen to more aggressively swap out keys after each
request
* Added ability to save/load configs by name
- New PUT method for config allows changing config with specified name
- New methods in js controller to handle loading/saving of configs
* Result formatting and removal of unused elements
- Fixed question section formatting from results page (added appropriate
padding and made questions styled as italic)
- Removed user agent display from main config settings
* Minor change to save config button label (now "Save As...")
* Fixed issue with "de-pickling" of flask session
Having a gitignore-everything ("*") file within a flask session folder seems to cause a
weird bug where the state of the app becomes unusable from continuously
trying to prune files listed in the gitignore (and it can't prune '*').
* Switched to pickling saved configs
* Updated ad/sponsored content filter and conf naming
Configs are now named with a .conf extension to allow for easier manual
cleanup/modification of named config files
Sponsored content now removed by basic string matching of span content
* Version bump to 0.2.0
* Fixed request.send return style
* Moved custom conf files to their own directory
* Refactored whoogle session mgmt
Now allows a fallback "default" session to be used if a user's browser
is blocking cookies
* Reworked pytest client fixture to support new session mgmt
* Added better multilingual support, updated filter
Results page now includes method for switching to "All Languages" from
whichever language is specified as the primary in the config (see #74).
Also removes the non-Whoogle links from the page footer, leaving only
the page navigation controls
Added support for the date range filter on the results page, though I'd
still recommend using the ":past <unit>" query instead.
* Removed no-cache enforcement, minor styling/formatting improvements
* Improving ad filtering for non-English languages
* Added footer to results page
Added enter key submit on results page
Added results type carryover for subsequent searches on results page
Removed redundant header on image search results
Basic autocomplete/search suggestion functionality added
* Adds new GET and POST routes for '/autocomplete' that accept a string query and returns an array of suggestions
* Adds new autoscript.js file for handling queries on the main page and results view
* Updated requests class to include autocomplete method
* Updated opensearch template to handle search suggestions
* Added header template to allow for autocomplete on results view
* Updated readme to mention autocomplete feature
Ben Busby