[[!meta title="Developer Weekend (September 2017)"]]
[[!meta author="Daniel Silverstone"]]
[[!meta date="2017-09-00 17:30:00"]]
* Michael Drake
* Vincent Sanders
* Daniel Silverstone
* John-Mark Bell
* Chris Young
* Logging merge for main NetSurf
* Daniel added verbose and normal logging options
* Vince added documentation for nslog in netsurf
* Vince fixed a whole pile of horrible compile problems with nslog on atari etc.
* Daniel fixed the test suite to work with the new logging options
* Triage of bug tracker to generate further topics for Saturday
* Michael found a couple of 10+ year old feature requests assigned to him, and decided
it was time to implement them, so he got started on:
* [`#1949` Search option for hotlist](http://bugs.netsurf-browser.org/mantis/view.php?id=1949)
* [`#1954` No way to search global history](http://bugs.netsurf-browser.org/mantis/view.php?id=1954)
* Daniel fixed [`#2549`](http://bugs.netsurf-browser.org/mantis/view.php?id=2549)
* Daniel reviewed [`#2537`](http://bugs.netsurf-browser.org/mantis/view.php?id=2537)
and decided that it wasn't worth trying to fix now. It's reported against
3.6, the reported issue is fixed, but the underlying inability to interact
with buttons is not.
* Daniel and vince reviewed [`#2555`](http://bugs.netsurf-browser.org/mantis/view.php?id=2555)
and decided François had already fixed it.
* Daniel reviewed [`#2544`](http://bugs.netsurf-browser.org/mantis/view.php?id=2544)
and decided it had always worked, closed it can't reproduce.
* Daniel tried to reproduce [`#2532`](http://bugs.netsurf-browser.org/mantis/view.php?id=2532)
but failed, so asked the submitter to try again.
* Daniel looked at:
and closed them unable-to-reproduce.
* Daniel reviewed [`#526`](http://bugs.netsurf-browser.org/mantis/view.php?id=526)
and assigned it to Michael
* Vince reviewed [`#2550`](http://bugs.netsurf-browser.org/mantis/view.php?id=2550)
and closed it unable-to-reproduce.
* Daniel and Vince worked through the following, closing most of them...
* [`#518`](http://bugs.netsurf-browser.org/mantis/view.php?id=518) (Actually we resolved this one)
* [`#469`](http://bugs.netsurf-browser.org/mantis/view.php?id=469) (Actually we resolved this one)
* [`#520`](http://bugs.netsurf-browser.org/mantis/view.php?id=520) (Actually we resolved this one)
* [`#1916`](http://bugs.netsurf-browser.org/mantis/view.php?id=1916) (We won'tfix'd this)
* [`#1907`](http://bugs.netsurf-browser.org/mantis/view.php?id=1907) (We trolled jmb)
* [`#1879`](http://bugs.netsurf-browser.org/mantis/view.php?id=1879) (Resolved)
* What frontends do we demote?
* See below
* Administrivia - Phoenix's disks are tired (even SMART said so)
* We'd like to know where the society bank account stands, for general
* We'd like to buy a pair of WD Gold 1TB drives (rough cost 140 UKP total)
so that Phoenix isn't running on pre-fail age drives.
* We investigated cloud/hosted servers, sadly it's super-expensive compared
to some new disks for the free hosting we have.
* Decision was: Michael, Vince, and Daniel will share the cost if the
society lacks the funds. Vince will buy drives and begin the replacement
process. Ideally ASAP.
* Discussion around forms, removing gadgets, etc.
* See below
* Auto-scroll to last position during history navigation
* See below
* Breaking changes and 4.x
* We have defined 'major' changes as anything large (e.g. new layout would
* Also 'major' changes include on-disk format changes. This means that
we'll forward-migrate where possible, but make no guarantees about going
backwards. Migration will be explicit and cancellable on startup.
* Content migrations must be such that if you run an old version, it won't
pick up old content by default.
* What's left to go in to 3.7
* Michael's searching for hotlist (treeview search)
* Vince's fixes for bitmaps in local history
* History navigation keeping scroll offsets
* Logging rework fallout
* General memory leak fixes where possible
* Aiming for 3.7 release handle crankage around Saturday September 23rd.
* URLdb is a mess. We knew that, there's nothing new there.
* We'd like to replace it, in its entirety, including on-disk formats.
* It should be split into a proper cookie jar, and a metadata store keyed
* For the cookie jar, there is an RFC for cookie handling, lots of useful
information and links on
* For the "URL database" we need to store arbitrary and changeable sets
of key-to-value mappings, some of which need to persist, some of which
are to be RAM-only.
* If we add scope to the 'key' such that we can have localstorage as a
scope then we can put local storage <https://www.w3.org/TR/webstorage/>
into the same DB.
* Potential implementation strategy:
* Use a journalling approach (where the entries are
create/change and delete commands) storing the journal as the URLdb
* The primary key into the DB is based on `nsurl`.
* We will reuse the `resource:` scheme for "internal" storage.
(e.g. session state, disk cache indices etc)
* The "value set" is a bag of triples. (`scope`, `key`, `value`) where
the scope is one of an enumeration (e.g. `localStorage` or `urldb`)
the key is a simple utf-8 string, and the value is an arbitrary set
of bytes (up to 4GB). (thus is stored as bytes+u32_length).
* We need the in-memory representation to support the following
1. Counting of key,value pairs in a given scope for a given url
2. Indexing a given (url,scope) pair by integer stably
3. Indexing a given (url,scope) pair by key name
4. Construction of journal entries (on changes)
5. Completion of url strings
6. Distinguish between content which can be persisted and that
which should not.
* The "clean" writeout should ensure that everything necessary for the
session restoration gets written out first, and then a marker is put down
which the readback code can use to kick off an async load of the rest of
* File format should be binary, and be robust (length, kind, checksum,
* Plausible things to layer on this generic storage are:
* URLdb replacement
* Session management
* Cookie jar
* File-based cache backend (indices)
* User options.
* Multiplicity (no single core)
* Proper testing for this.
* We do all the other stuff first, since it all has JS implications
* We need to update the IDLs and include more parts (e.g. LocalStorage,
* nsgenbind - should it do more automatic generation? If so how?
* Fix nsgenbind's `#file` `#line` stuff so that debugging works better.
* Fill out more of the bindings.
* So much of libdom needs enhancing too.
Chris Young still maintaining it. Do we want to add OS3 to the CI?
> Keep '1st class'
No maintainer, toolchain is awkward, blobby, unpleasant for CI. Needs a
scrub and refactor. We're not sure our builds are valid for Coldfire.
> Demote '2nd class' (no CI builds) after 3.7
Intermittent maintenance. Haiku builder is a pain in the arse, is the only
blocker for Java 8.
> Keep `1st class` for now, consult with Pulkomandy and mmu_man about JDK-8
Currently 2nd class, Libraries only. Sven came back, but made a messy branch
which was too hard to review/merge. Then he vanished once more. Is also a JDK
8 limiting blocker, as it is 2nd class, we'd just drop from CI entirely if it
was the only blocker.
> Keep `2nd class` for now.
Currently 1st class, boring, needs keyboard shortcuts etc. Hotlist support,
blahblahblah but frankly "just works™" for the most part (Raw linux framebuffer
doesn't work, but that's `libnsfb` really)
> Keep `1st class` for now.
Currently 1st class, basically works, lacks keyboard shortcuts and polish in
> Keep `1st class` for now.
Currently 1st class, doesn't tend to cause problems, needs further work for
a test driver etc. No proper maintainer right now.
> Keep `1st class` for now.
No maintainer, none of the core devs actually understand it any more, it's a
pain in the arse. Sadly most of our feedback comes from RISC OS users. For
now we do a best-effort on keeping it first class.
> Keep `1st class` for now.
Minimally usable, but there are problems with XP (though do we care?) some
issues around certificate bundles, storing of userdata, etc. Needs better
configuration, translations, etc.
> Keep `1st class` for now.
Forms and gadgets
The issues as we see them are that basically it's impossible to make small
fixes because whenever something is changed, it goes horribly wrong somewhere
else. Usually manifesting in double-frees or similar given the particularly
baroque ownership semantics (or lack thereof).
Rather than hacking it all out and starting from scratch, we proposed:
1. Making sure the gadgets update the DOM
* Gadget types: `HIDDEN`, `TEXTBOX`, `RADIO`, `CHECKBOX`, `SELECT`,
`TEXTAREA`, `IMAGE`, `PASSWORD`, `SUBMIT`, `RESET`, `FILE`, `BUTTON`
* It's likely `HIDDEN` can go away as soon as 2 is done.
* Validate that all these input kinds are supported in the libdom binding
* **TODO** Need to work out how to handle the reset event
2. Making the form submission read from the DOM
* Turns out Daniel wrote that in 2014. Oops. Nothing to do here.
3. Make the DOM trigger the submission (may need onclick or other event stuff)
* **TODO** First up read
<https://www.w3.org/TR/html5/forms.html#concept-fs-action> and implement
the form action, method, etc attributes onto the input element etc.
This may need libdom tweaks for the attributes and the novalidate stuff.
* **TODO** Rework the libdom submit stuff to use an internal event name
because the `submit` event is for non-`form.submit()` stuff:
* **TODO** Now determine how to implement the above bits neatly so that
we trigger submission properly.
4. Make any reader of the gadgets use the DOM
5. Work out how to remove more of this...
**Sadly it turned out not to be that simple**
1. Using <https://www.w3.org/TR/html5/forms.html> go through each of
`HTMLFormElement`, `HTMLInputElement`, `HTMLButtonElement`,
`HTMLTextAreaElement`, etc. and implement their functionality properly in
libdom. This may involve adding `RadioNodeList` if we don't have that yet.
It is likely to also involve needing to write tests which may or may not be
unpleasant to do.
2. Repeat through that, ensuring that the NetSurf core functionality uses the
new `libdom` interfaces properly. This will involve handling events, firing
events, etc. as needed, as well as handling form data set generation (though
that is mostly done, just may need tweaking wrt. which interfaces it uses).
1. It's possible that we will need to work out a clean way for nodes in the
DOM to listen to events. E.g. how does `dom_html_text_area_element` know
when its `textContent` has changed? It needs to know, so that it can redo
its `raw_value` and then fire appropriate events...
2. This will, if done properly, introduce a significant amount of intelligence
into libdom about how values are represented, perhaps we will defer the
validation stuff for now.
3. This will introduce more memory consumption for forms, though that is
unlikely to be a large amount.
4. Select elements and option elements may become more complicated as a result,
in order to cope with the reset algorithm. Also they will have to listen
to their subtrees and update themselves if necessary.
5. Note that `form[name]` is a horrific horrific mess and may terrify the most
hardened of cynics. Also note it's probably really heavily used. Oh Gods.
This horror is implemented via a _past names map_ which is per-form.
6. Note that `form.elements` is not an `HTMLCollection` It's an
`HTMLFormControlsCollection` which has a subtly different `.namedItem()`
7. Input elements can set a form owner directly using the `form` attribute
(note not content attribute, dunno how that applies in `libdom`, perhaps
we have to retain form owner somehow as an internal reference?)
8. Radio input elements introduce the concept of a radio button group which
might need some fun calculation because it can happen in things without
9. File input elements might have a horrific value attribute involving
nasty `C:\fakepath\` type bullshit. ARGH
10. Image input elements might be horrific, but we might be able to skip a lot
of the "delay load event" type crap by simply ignoring it and going lalala
11. Things like input value attribute changes might need extra support in the
layout engine to support things like caret movement.
12. File input elements have a `files` IDL attribute which returns the selected
set of filenames. Do we manage that list on libdom for funsies?
13. Input and Change events are not fired by the DOM directly, but by the UA's
input elements. This simplifies matters for us dramatically in the DOM but
may mean extra spots of checking in the client.
14. Form association is a pig, reread
Fortunately the `form` attribute appears to be readonly at the IDL level
and as such we might get away with ignoring the point I made in _7_ above.
15. Select element's length is not readonly, and it adjusts the underlying DOM
afaict. This horror, and associated gubbins is actually a reflection of
the same APIs on `HTMLOptionsCollection`. Argh.
16. Progress, Meter, etc. are not actually form elements
17. Keyset doesn't need to be implemented
18. TextAreas are a pain, but don't look overcomplex beyond the above.
19. FieldSet will need to be properly implemented
20. The 'disabled' attribute of things will need careful consideration because
it needs to flow through fieldsets etc. potentially. Perhaps fieldsets set
the disabled on the children instead, but that's not clear for resetting.
21. The Legend element may need implementing so that `form_owner` can be
presented to the UA.
22. The underlying DOM support for autofill might need implementing but frankly
can wait, we're not going to deal with that shizzle for now.
23. I recommend we don't do any of the constraint validation for now, including
minLength/maxLength stuff because it's a pig.
History navigation scroll
1. Update `struct history_entry` to contain a `last_offset` as a float
which is the proportion of the page scrolled (initialised to zero during
2. On navigation, if current is not NULL, store the `last_offset` into it
(note in `browser_window_navigate`)
3. On page completion, if the current history entry's `last_offset` is non-zero
then jump to that scroll location.
Since we have fragment handling already (and navigating to a fragment will mean
that the `last_offset` is zero) we will continue to function with fragments.
* Michael found and fixed a GTK core window issue where modifier keys were not
getting passed with mouse clicks (release), only with press. This prevented
Hotlist/Bookmarks entry editing.
* Daniel fixed a set of leaks from the logging layer.
We have chosen the next developer weekend to be .
It shall be at