[[!meta title="Developer Weekend (August 2019)"]]
[[!meta author="NetSurf Developers"]]
[[!meta date="2019-08-01 10:00:00"]]
* Michael Drake
* Vincent Sanders
* Daniel Silverstone
* John-Mark Bell
Outstanding work (from May)
* Driver - capability to "Click" on a specified piece of text (Daniel)
(Basically a way to say click button FOO) **DONE**
* Language support for resources (Vince)
* Listing of compiled-in surfaces (Vince)
We went over coverity output because we got upgraded. Fixed a few items along
Plenty of triage was done. In addition, we fixed:
* Coverity fix for LibCSS that makes the select hash insertion at start of non-empty list code more readable.
* Worked with Daniel on fixing NULL node hover crash on page with all content set to `visibility: hidden`.
* Much plotting and planning of libnslayout's text layout, with Daniel.
* Looked at browser window scale stuff with Vince and moved the scale handling for invalidate to the core, and updated the GTK front end.
* Setting up of frameworks for new libparagraph stuff.
* Updated duktape to version 2.4.0.
* Made local-history bitmaps scale with DPI
* Made iframe test on <https://test.netsurf-browser.org/html/iframe.html>
* Attempted to add support for core window `scroll_visible` to RISC OS front end.
- It didn't compile, so built the RISC OS toolchain and SDK, and then fixed the compilation error.
* Added support for scaling keyboard shortcuts to the Framebuffer front end.
* Optimised libnsfb's xrgb8888 surface rendering using a neat trick from Adrian Lees.
* Added support for scrolling core windows to the RISC OS front end.
* Fixed the RISC OS core window get_window_dimensions to handle toolbar height.
* Made all the core window callbacks return nserror instead of void.
- And updated all the front ends.
* Changed the core window interface for scrolling from scroll_visible to much simpler get/set scroll APIs.
* Reimplemented old scroll_visisble API in the core on top of the new set/get scroll and get_window_dimensions, using Daniel's algorithm from the GTK front end.
* Made the RISC OS local history window claim input focus when it opens.
* Fixed a couple of Duktape bindings for DOM element nodes that didn't check the return value of a LibDOM call.
* Suppressed dead store `scan-build` warnings in the CSS presentational hints gatherer.
* Wrote some simple iframe and image tests for bad SSL.
* Implemented the libparagraph stuff for creating and destroying contexts.
* Implemented the libparagraph stuff for adding content to contexts.
* Added the libparagraph stuff to the Makefile, to build it.
* Looked into crash with animated GIF redraw.
- Optimised box_coords when there's a float in the ancestors.
- Potentially fixed crash by short-circuiting object redraws before the HTML has had a layout pass.
* Looked at CSS styling stuff with Vince.
* A number of bug fixes
* Worked with Michael planning text layout
* Acquired a polyfill for `Array.from()` and fixed `consoleFormatter`.
* Fixed a typo (`BUTTOM`)
* Made it so that cURL fetches will abort early if possible
* Updated stacktrace handling to new duktape
* Added cookie setting support
* Wrote reload support into monkey driver/farmer
* Added a cookie test using JS and a new cookies CGI
* Various cleanups with `fetch_fdset()` vs `schedule_run()`
* Added extra diagnostics during monkeyfarmer shutdown
* Fixed the oldest bug! (Added scroll-visible to local history and added key
* Did similarly to treeview (scroll-visible)
* Added click support to monkey, monkeyfarmer, monkey-driver, and added a test
to netsurf-test which will use it.
* More robustification of the asyncio loop in monkeyfarmer.
* Added additional intuitive nav to local history
* Added a 'aborted-fetch' test (and fixed monkey_driver a bit)
* Migrated the callback for auth/ssl from llcache through hlcache to the users
of the hlcache. It should be functionally equivalent (except sub-fetches
no longer prompt but instead automatically fail like in other browsers).
* Designed an approach for shadow contents, fleshed it out, discussed it among
ourselves, discarded as being overcomplex. It is preserved below for posterity.
* Designed, with Vince primarily being the clever monkey, a new approach which
removed the query callbacks from the llcache and instead uses content generated
in the `about:` handler. We think this is cleaner.
* Combined `CONTENT_MSG_ERROR` and `CONTENT_MSG_ERRORCODE`
* Taught `browser_window` how to remember fetch parameters so that we can
restart fetches in the new query design. Split navigation in half to make
* Migrated SSL certificate chain storage to `browser_window` and allowed it
to store the chains for the completed fetch, leading to possibilities of
padlocks later down the line. In the process, fixed a long-standing bug
where serial numbers never worked.
* Excised the llcache query pathways and migrated them all the way up to the
* Tracked down a strange behaviour and discovered we didn't handle `PAUSED`
coming out when we tried to finish the parse. Added a fix for that.
* Added internal navigation concept to `browser_window` and migrated the
login handling inside it, using Vince's new auth query support in the `about:`
* Restored OpenBSD to functionality in the CI
* Splitting up of `browser.c`
* Removed `scale` from `_invalidate`.
* Removed scaling from `get_dimensions`.
* Re-fixed Haiku worker in CI.
* Cleared out leftover junk builds from the CI carrier.
* Tested Michael's RISC OS changes under RPCEmu.
* Completed `scale`ectomy in `browser_window_mouse_click()`, `browser_window_mouse_track()`, `browser_window_get_features()` `browser_window_set_scale()`. And updated all of the callers and all of the front ends.
- Now the front ends don't need to care about scale.
* wrote about scheme query handlers for auth and privacy
Thoughts on shadow contents -- We decided that this was overcomplex and CANCELLED THIS IDEA
In order to complete the work we started my migrating queryies all the way up
from the `llcache` to the `browser_window`, we need to come up with a way for
the `browser_window` to load and display a real content which will represent
the query to present to the user.
* **real content**: A real content if available (e.g. the current page when then
* **loading content**: The content which is currently loading into the window.
* **real loading content**: The _loading content_ which may trigger a query.
* **query**: The callback from the _real loading content_ which needs user
input to decide on the action for.
* **stored query**: A stored callback and private word for answering the
last _query_ which came for the _real loading content_.
* **shadow loading content**: A holding space for a _real loading content_
which should not give any events unless something else unblocks the query in
* **special navigation**: An attempt to navigate in such a way as it would
answer a _stored query_
The events we have to respond to are:
1. The query event coming into the `browser_window_callback`
2. Any attempt to `browser_window_navigate`
3. Any attempt to `browser_window_reload`
4. Any event coming from a _shadow loading content_
5. Destroying a `browser_window`
6. Any call to `browser_window_stop`
The states we might be in are:
1. **Not loading**: The normal state of being where we're not loading and as
such nothing interesting is going on.
2. **Normal loading**: The process of loading is happening, there is no
_shadow loading content_ and the _real loading content_ is making progress.
3. **Shadow loading**: The process of loading is paused, and a shadow page is
being loaded. This implies there's a _stored query_.
4. **Shadow loaded**: This is like the _not loading_ state, but there is a
_shadow loading content_ and a _stored query_ which mark it as different.
1. Moving out of _not loading_ happens as normal on any of the usual events
2. When _normal loading_ all the usual events behave in the current manner,
except if we receive a _query_. On receipt of a _query_ the _real loading
content_ is moved from the _loading content_ into the _shadow loading
content_, and a new request is created in the _loading content_ for the
query page with requisite parameters. Finally the _query_ parameters are
stored into the _stored query_. This moves us to _shadow loading_.
3. When _shadow loading_ and any of the navigation events occur, we first abort
the _loading content_ and move the _shadow loading content_ (which is the
_real loading content_) back into the _loading content_. Then we refuse the
_stored query_ which will abort the _real loading content_. Then we process
the navigation as before.
4. When loading completes when we're in _shadow loading_ then we proceed as
"normal" to replace the current content with the converted shadow document,
we fire the query at the frontend, and we enter _shadow loaded_.
5. When in _shadow loaded_ and normal navigation occurs, we behave essentially
like we would in _shadow loading_ and such an event occurred.
6. When in _shadow loaded_ and we receive a _special navigation_ instruction,
then we gather the information from the URL to answer the _stored query_,
then close the query page content, move the _shadow loading content_ back to
the _loading content_, and then clear the _stored query_. This transitions
us back into _normal loading_. Finally we send the answer to the query
which will kick off normal fetch procedures once more.
7. When in _shadow loaded_ or _shadow loading_ we receive any events from the
_shadow loading content_ we should assume that a decision was made in
another window and so we abort any _loading content_, or close any displayed
content, drop the _stored query_, put the _shadow loading content_ back into
the _loading content_, and return to _normal loading_
Replacement query proposal
In order to simplify matters we decided instead that we would unwind the
paused/callback mechanism and instead have the fetch stop with an appropriate
error code indicating the query, allow the browser window to deal with that
and then to restart the fetch from the top level.
To do so, the browser window will have to retain the post data etc. This
allows for us to also implement the when reloading a page which was posted,
query the user for whether or not to re-post. (but this is 'some future time')
Then when we receive a query form `CONTENT_MSG_ERROR` the browser window can
issue a special internal fetch for the query page it needs, passing the
requisite information. On `CONTENT_COMPLETE` it can then call the gui, giving
it a chance to chime in.
If a reload happens while we're on a special page, we redo the original fetch,
if we navigate to the special try-again, we redo the original fetch. If we
navigate to the special proceed, we perform the update as needed, and redo
the original fetch.
First, we need to clean up and unify `CONTENT_MSG_ERROR` and
This will allow us to later use this unified thing to communicate more useful
fetch errors upward toward the browser window.
We make `browser_window_navigate` store the fetch parameters so that
redoing the fetch is possible
We change the fetch/llcache/hlcache layers to always pass SSL certificate
chains upward when they are available.
And we change the `browser_window` to store that chain alongside both the
`current_content` and the `loading_content`.
We may add APIs to access these later.
We change the fetch/llcache/hlcache layers to remove the `QUERY` concept
entirely and to pass an appropriately structed `CONTENT_MSG_ERROR`.
We change the `browser_window` to consume that error and call the current
callbacks. The `proceed` callback will have to be implemented in
`browser_window` in a simplistic fashion.
At this point, all the plumbing is in place, but the current behaviour in terms
of GUI interaction and urldb remains the same.
We write a handler in about: to provide the query pages. This is MVP to be
able to get on with things. At this point we allow `browser_window` to have
the concept of a special fetch which does not update the fetch paramters, nor
the url bar etc. And when we stop a fetch because of the query errors, we
navigate to that about: handler using the special fetch mode, thus we'll end up
with both the new page *and* the GUI prompt still. Ideally we move the gui
prompt to be on `CONTENT_DONE` for the special load.
We handle the special `about:` URI which means 'proceed' and the one which
means 'try-again' as per the descriptions in the summary of this proposal.
Rework the gui/browser_window querying so that instead of proceed/cancel
it is try-again/take-answer/proceed.
Various sundry cleanups as a result of all of the above.
Statement of work
If at all possible, we'd like to see some of the following addressed before
the next developer weekend…
* Remove 401 login implementations from all frontends (Vince) **DONE**
* Resolve issues arising from introducing internal query pages (Daniel) **DONE**
* Implement any appropriate auto fill auth handlers
* Low level cache should own certificate data and persist with all other data
- Shouldn't be pushed down to browser window; browser window should request
it from the hlcache handle, which should get it from the llcache.
* Implement Site Information (padlock) and repurpose certificate error core
windows to be certificate viewing interfaces
* Continue styling the generated query pages.
* Review TODOs.
* Language support for resources (Vince)
* Listing of compiled-in surfaces (Vince)
* Write the English query page page messages first. (Vince) **DONE**
* Make a mailing list post about query pages. (Daniel) **DONE**
* Translations of all messages for the SSL/privacy internal query page.
* Translations of all messages for the authentication internal query page.
* Current CI tests (test-site-run)
- Three groups:
+ Tests that only fetch from our infrastructure.
+ Tests that fetch off third party servers and are fast to run.
+ Test that fetch from third party servers and long running.
- In Jenkins: Group the first two and make them trigger the third one.
* New tests
* Take the 10,000 sites list, split it up into blocks of 20, and run them as
* Four core VM, which gets 4 Jenkins jobs for running these tests.
* Experiment and see how it breaks down. Needs to be weekly (we can tweak it).
* Text layout
- Continue implementing. (Michael)
* 30 Nov / 1 Dec.
* Consider extending either side, depending on holidays.