path: root/developer-weekend/aug-2019.mdwn

[[!meta title="Developer Weekend (August 2019)"]]
[[!meta author="NetSurf Developers"]]
[[!meta date="2019-08-01 10:00:00"]]

[[!toc]]

Attendees
=========

* Michael Drake
* Vincent Sanders
* Daniel Silverstone

Apologies
=========

* John-Mark Bell

Outstanding work (from May)
============================

* Monkey
    * Driver - capability to "Click" on a specified piece of text (Daniel)
      (Basically a way to say click button FOO) **DONE**
* Framebuffer
    * Language support for resources (Vince)
    * Listing of compiled-in surfaces (Vince)

Topics
======


Discussions
===========


Frontends
=========

Coverity
========

We went over coverity output because we got upgraded.  Fixed a few items along
the way.

Bug Triage
==========

Plenty of triage was done.  In addition, we fixed:

* <https://bugs.netsurf-browser.org/mantis/view.php?id=2658>
* <https://bugs.netsurf-browser.org/mantis/view.php?id=2617>

Activity
========

Michael
-------

* Coverity fix for LibCSS that makes the select hash insertion at start of non-empty list code more readable.
* Worked with Daniel on fixing NULL node hover crash on page with all content set to `visibility: hidden`.
* Much plotting and planning of libnslayout's text layout, with Daniel.
* Looked at browser window scale stuff with Vince and moved the scale handling for invalidate to the core, and updated the GTK front end.
* Setting up of frameworks for new libparagraph stuff.
* Updated duktape to version 2.4.0.
* Made local-history bitmaps scale with DPI
* Made iframe test on <https://test.netsurf-browser.org/html/iframe.html>
* Attempted to add support for core window `scroll_visible` to RISC OS front end.
  - It didn't compile, so built the RISC OS toolchain and SDK, and then fixed the compilation error.
* Added support for scaling keyboard shortcuts to the Framebuffer front end.
* Optimised libnsfb's xrgb8888 surface rendering using a neat trick from Adrian Lees.
* Added support for scrolling core windows to the RISC OS front end.
* Fixed the RISC OS core window get_window_dimensions to handle toolbar height.
* Made all the core window callbacks return nserror instead of void.
  - And updated all the front ends.
* Changed the core window interface for scrolling from scroll_visible to much simpler get/set scroll APIs.
* Reimplemented old scroll_visisble API in the core on top of the new set/get scroll and get_window_dimensions, using Daniel's algorithm from the GTK front end.
* Made the RISC OS local history window claim input focus when it opens.
* Fixed a couple of Duktape bindings for DOM element nodes that didn't check the return value of a LibDOM call.
* Suppressed dead store `scan-build` warnings in the CSS presentational hints gatherer.
* Wrote some simple iframe and image tests for bad SSL.
* Implemented the libparagraph stuff for creating and destroying contexts.
* Implemented the libparagraph stuff for adding content to contexts.
* Added the libparagraph stuff to the Makefile, to build it.
* Looked into crash with animated GIF redraw.
  - Optimised box_coords when there's a float in the ancestors. 
  - Potentially fixed crash by short-circuiting object redraws before the HTML has had a layout pass.
* Looked at CSS styling stuff with Vince.

Daniel
------

* A number of bug fixes
* Worked with Michael planning text layout
* Acquired a polyfill for `Array.from()` and fixed `consoleFormatter`.
* Fixed a typo (`BUTTOM`)
* Made it so that cURL fetches will abort early if possible
* Updated stacktrace handling to new duktape
* Added cookie setting support
* Wrote reload support into monkey driver/farmer
* Added a cookie test using JS and a new cookies CGI
* Various cleanups with `fetch_fdset()` vs `schedule_run()`
* Added extra diagnostics during monkeyfarmer shutdown
* Fixed the oldest bug!  (Added scroll-visible to local history and added key
  navigation)
* Did similarly to treeview (scroll-visible)
* Added click support to monkey, monkeyfarmer, monkey-driver, and added a test
  to netsurf-test which will use it.
* More robustification of the asyncio loop in monkeyfarmer.
* Added additional intuitive nav to local history
* Added a 'aborted-fetch' test (and fixed monkey_driver a bit)
* Migrated the callback for auth/ssl from llcache through hlcache to the users
  of the hlcache.  It should be functionally equivalent (except sub-fetches
  no longer prompt but instead automatically fail like in other browsers).
* Designed an approach for shadow contents, fleshed it out, discussed it among
  ourselves, discarded as being overcomplex.  It is preserved below for posterity.
* Designed, with Vince primarily being the clever monkey, a new approach which
  removed the query callbacks from the llcache and instead uses content generated
  in the `about:` handler.  We think this is cleaner.
* Combined `CONTENT_MSG_ERROR` and `CONTENT_MSG_ERRORCODE`
* Taught `browser_window` how to remember fetch parameters so that we can
  restart fetches in the new query design.  Split navigation in half to make
  that clean.
* Migrated SSL certificate chain storage to `browser_window` and allowed it
  to store the chains for the completed fetch, leading to possibilities of
  padlocks later down the line.  In the process, fixed a long-standing bug
  where serial numbers never worked.
* Excised the llcache query pathways and migrated them all the way up to the
  `browser_window`
* Tracked down a strange behaviour and discovered we didn't handle `PAUSED`
  coming out when we tried to finish the parse.  Added a fix for that.
* Added internal navigation concept to `browser_window` and migrated the
  login handling inside it, using Vince's new auth query support in the `about:`
  fetcher.

Vince
-----

* Restored OpenBSD to functionality in the CI
* Splitting up of `browser.c`
* Removed `scale` from `_invalidate`.
* Removed scaling from `get_dimensions`.
* Re-fixed Haiku worker in CI.
* Cleared out leftover junk builds from the CI carrier.
* Tested Michael's RISC OS changes under RPCEmu.
* Completed `scale`ectomy in `browser_window_mouse_click()`, `browser_window_mouse_track()`, `browser_window_get_features()` `browser_window_set_scale()`.  And updated all of the callers and all of the front ends.
  - Now the front ends don't need to care about scale.
* wrote about scheme query handlers for auth and privacy


Thoughts on shadow contents -- We decided that this was overcomplex and CANCELLED THIS IDEA
===========================================================================================

In order to complete the work we started my migrating queryies all the way up
from the `llcache` to the `browser_window`, we need to come up with a way for
the `browser_window` to load and display a real content which will represent
the query to present to the user.

Terms
-----

* **real content**: A real content if available (e.g. the current page when then
  navigating away).
* **loading content**: The content which is currently loading into the window.
* **real loading content**: The _loading content_ which may trigger a query.
* **query**: The callback from the _real loading content_ which needs user
  input to decide on the action for.
* **stored query**: A stored callback and private word for answering the
  last _query_ which came for the _real loading content_.
* **shadow loading content**: A holding space for a _real loading content_
  which should not give any events unless something else unblocks the query in
  another window.
* **special navigation**: An attempt to navigate in such a way as it would
  answer a _stored query_

Events
------

The events we have to respond to are:

1. The query event coming into the `browser_window_callback`
2. Any attempt to `browser_window_navigate`
3. Any attempt to `browser_window_reload`
4. Any event coming from a _shadow loading content_
5. Destroying a `browser_window`
6. Any call to `browser_window_stop`

States
------

The states we might be in are:

1. **Not loading**: The normal state of being where we're not loading and as
   such nothing interesting is going on.
2. **Normal loading**: The process of loading is happening, there is no
   _shadow loading content_ and the _real loading content_ is making progress.
3. **Shadow loading**: The process of loading is paused, and a shadow page is
   being loaded.  This implies there's a _stored query_.
4. **Shadow loaded**: This is like the _not loading_ state, but there is a
   _shadow loading content_ and a _stored query_ which mark it as different.

State transitions
-----------------

1. Moving out of _not loading_ happens as normal on any of the usual events
2. When _normal loading_ all the usual events behave in the current manner,
   except if we receive a _query_.  On receipt of a _query_ the _real loading
   content_ is moved from the _loading content_ into the _shadow loading
   content_, and a new request is created in the _loading content_ for the
   query page with requisite parameters.  Finally the _query_ parameters are
   stored into the _stored query_.  This moves us to _shadow loading_.
3. When _shadow loading_ and any of the navigation events occur, we first abort
   the _loading content_ and move the _shadow loading content_ (which is the
   _real loading content_) back into the _loading content_.  Then we refuse the
   _stored query_ which will abort the _real loading content_.  Then we process
   the navigation as before.
4. When loading completes when we're in _shadow loading_  then we proceed as
   "normal" to replace the current content with the converted shadow document,
   we fire the query at the frontend, and we enter _shadow loaded_.
5. When in _shadow loaded_ and normal navigation occurs, we behave essentially
   like we would in _shadow loading_ and such an event occurred.
6. When in _shadow loaded_ and we receive a _special navigation_ instruction,
   then we gather the information from the URL to answer the _stored query_,
   then close the query page content, move the _shadow loading content_ back to
   the _loading content_, and then clear the _stored query_.  This transitions
   us back into _normal loading_.  Finally we send the answer to the query
   which will kick off normal fetch procedures once more.
7. When in _shadow loaded_ or _shadow loading_ we receive any events from the
   _shadow loading content_ we should assume that a decision was made in
   another window and so we abort any _loading content_, or close any displayed
   content, drop the _stored query_, put the _shadow loading content_ back into
   the _loading content_, and return to _normal loading_

Replacement query proposal
==========================

In order to simplify matters we decided instead that we would unwind the
paused/callback mechanism and instead have the fetch stop with an appropriate
error code indicating the query, allow the browser window to deal with that
and then to restart the fetch from the top level.

To do so, the browser window will have to retain the post data etc.  This
allows for us to also implement the when reloading a page which was posted,
query the user for whether or not to re-post. (but this is 'some future time')

Then when we receive a query form `CONTENT_MSG_ERROR` the browser window can
issue a special internal fetch for the query page it needs, passing the
requisite information.  On `CONTENT_COMPLETE` it can then call the gui, giving
it a chance to chime in.

If a reload happens while we're on a special page, we redo the original fetch,
if we navigate to the special try-again, we redo the original fetch.  If we
navigate to the special proceed, we perform the update as needed, and redo
the original fetch.

Step one
--------

First, we need to clean up and unify `CONTENT_MSG_ERROR` and
`CONTENT_MSG_ERRORCODE`.

This will allow us to later use this unified thing to communicate more useful
fetch errors upward toward the browser window.

Step two
--------

We make `browser_window_navigate` store the fetch parameters so that
redoing the fetch is possible

Step three
----------

We change the fetch/llcache/hlcache layers to always pass SSL certificate
chains upward when they are available.

And we change the `browser_window` to store that chain alongside both the
`current_content` and the `loading_content`.

We may add APIs to access these later.

Step four
---------

We change the fetch/llcache/hlcache layers to remove the `QUERY` concept
entirely and to pass an appropriately structed `CONTENT_MSG_ERROR`.

We change the `browser_window` to consume that error and call the current
callbacks.  The `proceed` callback will have to be implemented in
`browser_window` in a simplistic fashion.

At this point, all the plumbing is in place, but the current behaviour in terms
of GUI interaction and urldb remains the same.

Step five
---------

We write a handler in about: to provide the query pages.  This is MVP to be
able to get on with things.  At this point we allow `browser_window` to have
the concept of a special fetch which does not update the fetch paramters, nor
the url bar etc.  And when we stop a fetch because of the query errors, we
navigate to that about: handler using the special fetch mode, thus we'll end up
with both the new page *and* the GUI prompt still.  Ideally we move the gui
prompt to be on `CONTENT_DONE` for the special load.

Step six
--------

We handle the special `about:` URI which means 'proceed' and the one which
means 'try-again' as per the descriptions in the summary of this proposal.

Step seven
----------

Rework the gui/browser_window querying so that instead of proceed/cancel
it is try-again/take-answer/proceed.

Step eight
----------

Various sundry cleanups as a result of all of the above.

Statement of work
=================

If at all possible, we'd like to see some of the following addressed before
the next developer weekend…

* General
    * Remove 401 login implementations from all frontends (Vince) **DONE**
    * Resolve issues arising from introducing internal query pages (Daniel) **DONE**
    * Implement any appropriate auto fill auth handlers
    * Low level cache should own certificate data and persist with all other data
      - Shouldn't be pushed down to browser window; browser window should request
        it from the hlcache handle, which should get it from the llcache.
    * Implement Site Information (padlock) and repurpose certificate error core
      windows to be certificate viewing interfaces
    * Continue styling the generated query pages.
    * Review TODOs.
* Framebuffer
    * Language support for resources (Vince)
    * Listing of compiled-in surfaces (Vince)
* Internationalisation
    * Write the English query page page messages first. (Vince) **DONE**
    * Make a mailing list post about query pages. (Daniel) **DONE**
    * Translations of all messages for the SSL/privacy internal query page.
    * Translations of all messages for the authentication internal query page.
* Current CI tests (test-site-run)
  - Three groups:
    + Tests that only fetch from our infrastructure.
    + Tests that fetch off third party servers and are fast to run.
    + Test that fetch from third party servers and long running.
  - In Jenkins: Group the first two and make them trigger the third one.
* New tests
  * Take the 10,000 sites list, split it up into blocks of 20, and run them as
    independent tests.
  * Four core VM, which gets 4 Jenkins jobs for running these tests.
  * Experiment and see how it breaks down. Needs to be weekly (we can tweak it).
* Text layout
  - Continue implementing. (Michael)

Next time
=========

* Manchester
* 30 Nov / 1 Dec.
* Consider extending either side, depending on holidays.
* changed from Cambridge because of Daniel and Micahel availability