[[!meta title="Developer Weekend (September 2017)"]] [[!meta author="Daniel Silverstone"]] [[!meta date="2017-09-00 17:30:00"]] [[!toc]] Attendees ========= * Michael Drake * Vincent Sanders * Daniel Silverstone Apologies ========= * John-Mark Bell * Chris Young Topics ====== * Logging merge for main NetSurf * Daniel added verbose and normal logging options * Vince added documentation for nslog in netsurf * Vince fixed a whole pile of horrible compile problems with nslog on atari etc. * Daniel fixed the test suite to work with the new logging options * Triage of bug tracker to generate further topics for Saturday * Michael found a couple of 10+ year old feature requests assigned to him, and decided it was time to implement them, so he got started on: * [`#1949` Search option for hotlist](http://bugs.netsurf-browser.org/mantis/view.php?id=1949) * [`#1954` No way to search global history](http://bugs.netsurf-browser.org/mantis/view.php?id=1954) * Daniel fixed [`#2549`](http://bugs.netsurf-browser.org/mantis/view.php?id=2549) * Daniel reviewed [`#2537`](http://bugs.netsurf-browser.org/mantis/view.php?id=2537) and decided that it wasn't worth trying to fix now. It's reported against 3.6, the reported issue is fixed, but the underlying inability to interact with buttons is not. * Daniel and vince reviewed [`#2555`](http://bugs.netsurf-browser.org/mantis/view.php?id=2555) and decided François had already fixed it. * Daniel reviewed [`#2544`](http://bugs.netsurf-browser.org/mantis/view.php?id=2544) and decided it had always worked, closed it can't reproduce. * Daniel tried to reproduce [`#2532`](http://bugs.netsurf-browser.org/mantis/view.php?id=2532) but failed, so asked the submitter to try again. * Daniel looked at: * [`#2464`](http://bugs.netsurf-browser.org/mantis/view.php?id=2464) * [`#2482`](http://bugs.netsurf-browser.org/mantis/view.php?id=2482) * [`#2417`](http://bugs.netsurf-browser.org/mantis/view.php?id=2417) and closed them unable-to-reproduce. * Daniel reviewed [`#526`](http://bugs.netsurf-browser.org/mantis/view.php?id=526) and assigned it to Michael * Vince reviewed [`#2550`](http://bugs.netsurf-browser.org/mantis/view.php?id=2550) and closed it unable-to-reproduce. * Daniel and Vince worked through the following, closing most of them... * [`#1885`](http://bugs.netsurf-browser.org/mantis/view.php?id=1885) * [`#1886`](http://bugs.netsurf-browser.org/mantis/view.php?id=1886) * [`#1889`](http://bugs.netsurf-browser.org/mantis/view.php?id=1889) * [`#420`](http://bugs.netsurf-browser.org/mantis/view.php?id=420) * [`#434`](http://bugs.netsurf-browser.org/mantis/view.php?id=434) * [`#518`](http://bugs.netsurf-browser.org/mantis/view.php?id=518) (Actually we resolved this one) * [`#469`](http://bugs.netsurf-browser.org/mantis/view.php?id=469) (Actually we resolved this one) * [`#520`](http://bugs.netsurf-browser.org/mantis/view.php?id=520) (Actually we resolved this one) * [`#522`](http://bugs.netsurf-browser.org/mantis/view.php?id=522) * [`#505`](http://bugs.netsurf-browser.org/mantis/view.php?id=505) * [`#1916`](http://bugs.netsurf-browser.org/mantis/view.php?id=1916) (We won'tfix'd this) * [`#1907`](http://bugs.netsurf-browser.org/mantis/view.php?id=1907) (We trolled jmb) * [`#1879`](http://bugs.netsurf-browser.org/mantis/view.php?id=1879) (Resolved) * [`#485`](http://bugs.netsurf-browser.org/mantis/view.php?id=485) * What frontends do we demote? * See below * Administrivia - Phoenix's disks are tired (even SMART said so) * We'd like to know where the society bank account stands, for general resources. * We'd like to buy a pair of WD Gold 1TB drives (rough cost 140 UKP total) so that Phoenix isn't running on pre-fail age drives. * We investigated cloud/hosted servers, sadly it's super-expensive compared to some new disks for the free hosting we have. * Decision was: Michael, Vince, and Daniel will share the cost if the society lacks the funds. Vince will buy drives and begin the replacement process. Ideally ASAP. * Discussion around forms, removing gadgets, etc. * See below * Auto-scroll to last position during history navigation * See below * Breaking changes and 4.x * We have defined 'major' changes as anything large (e.g. new layout would be major) * Also 'major' changes include on-disk format changes. This means that we'll forward-migrate where possible, but make no guarantees about going backwards. Migration will be explicit and cancellable on startup. * Content migrations must be such that if you run an old version, it won't pick up old content by default. * What's left to go in to 3.7 * Michael's searching for hotlist (treeview search) * Vince's fixes for bitmaps in local history * History navigation keeping scroll offsets * Logging rework fallout * General memory leak fixes where possible * Aiming for 3.7 release handle crankage around Saturday September 23rd. * URLdb/Cookies * URLdb is a mess. We knew that, there's nothing new there. * We'd like to replace it, in its entirety, including on-disk formats. * It should be split into a proper cookie jar, and a metadata store keyed by URL. * For the cookie jar, there is an RFC for cookie handling, lots of useful information and links on * For the "URL database" we need to store arbitrary and changeable sets of key-to-value mappings, some of which need to persist, some of which are to be RAM-only. * If we add scope to the 'key' such that we can have localstorage as a scope then we can put local storage into the same DB. * Potential implementation strategy: * Use a journalling approach (where the entries are create/change and delete commands) storing the journal as the URLdb replacement. * The primary key into the DB is based on `nsurl`. * We will reuse the `resource:` scheme for "internal" storage. (e.g. session state, disk cache indices etc) * The "value set" is a bag of triples. (`scope`, `key`, `value`) where the scope is one of an enumeration (e.g. `localStorage` or `urldb`) the key is a simple utf-8 string, and the value is an arbitrary set of bytes (up to 4GB). (thus is stored as bytes+u32_length). * We need the in-memory representation to support the following efficiently: 1. Counting of key,value pairs in a given scope for a given url 2. Indexing a given (url,scope) pair by integer stably 3. Indexing a given (url,scope) pair by key name 4. Construction of journal entries (on changes) 5. Completion of url strings 6. Distinguish between content which can be persisted and that which should not. * The "clean" writeout should ensure that everything necessary for the session restoration gets written out first, and then a marker is put down which the readback code can use to kick off an async load of the rest of the data. * File format should be binary, and be robust (length, kind, checksum, payload). * Plausible things to layer on this generic storage are: * URLdb replacement * Session management * LocalStorage * Cookie jar * File-based cache backend (indices) * User options. * Multiplicity (no single core) * Proper testing for this. * The "What's Next?" of Javascript * We do all the other stuff first, since it all has JS implications * We need to update the IDLs and include more parts (e.g. LocalStorage, CSSOM, etc) * nsgenbind - should it do more automatic generation? If so how? * Fix nsgenbind's `#file` `#line` stuff so that debugging works better. * Fill out more of the bindings. * So much of libdom needs enhancing too. Frontends ========= Amiga ----- Chris Young still maintaining it. Do we want to add OS3 to the CI? > Keep '1st class' Atari ----- No maintainer, toolchain is awkward, blobby, unpleasant for CI. Needs a scrub and refactor. We're not sure our builds are valid for Coldfire. > Demote '2nd class' (no CI builds) after 3.7 BeOS ---- Intermittent maintenance. Haiku builder is a pain in the arse, is the only blocker for Java 8. > Keep `1st class` for now, consult with Pulkomandy and mmu_man about JDK-8 Cocoa ----- Currently 2nd class, Libraries only. Sven came back, but made a messy branch which was too hard to review/merge. Then he vanished once more. Is also a JDK 8 limiting blocker, as it is 2nd class, we'd just drop from CI entirely if it was the only blocker. > Keep `2nd class` for now. Framebuffer ----------- Currently 1st class, boring, needs keyboard shortcuts etc. Hotlist support, blahblahblah but frankly "just works™" for the most part (Raw linux framebuffer doesn't work, but that's `libnsfb` really) > Keep `1st class` for now. GTK --- Currently 1st class, basically works, lacks keyboard shortcuts and polish in general. > Keep `1st class` for now. Monkey ------ Currently 1st class, doesn't tend to cause problems, needs further work for a test driver etc. No proper maintainer right now. > Keep `1st class` for now. RISC OS ------- No maintainer, none of the core devs actually understand it any more, it's a pain in the arse. Sadly most of our feedback comes from RISC OS users. For now we do a best-effort on keeping it first class. > Keep `1st class` for now. Windows ------- Minimally usable, but there are problems with XP (though do we care?) some issues around certificate bundles, storing of userdata, etc. Needs better configuration, translations, etc. > Keep `1st class` for now. Forms and gadgets ================= The issues as we see them are that basically it's impossible to make small fixes because whenever something is changed, it goes horribly wrong somewhere else. Usually manifesting in double-frees or similar given the particularly baroque ownership semantics (or lack thereof). Rather than hacking it all out and starting from scratch, we proposed: 1. Making sure the gadgets update the DOM * Gadget types: `HIDDEN`, `TEXTBOX`, `RADIO`, `CHECKBOX`, `SELECT`, `TEXTAREA`, `IMAGE`, `PASSWORD`, `SUBMIT`, `RESET`, `FILE`, `BUTTON` * It's likely `HIDDEN` can go away as soon as 2 is done. * Validate that all these input kinds are supported in the libdom binding * **TODO** Need to work out how to handle the reset event 2. Making the form submission read from the DOM * Turns out Daniel wrote that in 2014. Oops. Nothing to do here. 3. Make the DOM trigger the submission (may need onclick or other event stuff) * **TODO** First up read and implement the form action, method, etc attributes onto the input element etc. This may need libdom tweaks for the attributes and the novalidate stuff. * **TODO** Rework the libdom submit stuff to use an internal event name because the `submit` event is for non-`form.submit()` stuff: * **TODO** Now determine how to implement the above bits neatly so that we trigger submission properly. 4. Make any reader of the gadgets use the DOM 5. Work out how to remove more of this... **Sadly it turned out not to be that simple** New approach ------------ 1. Using go through each of `HTMLFormElement`, `HTMLInputElement`, `HTMLButtonElement`, `HTMLTextAreaElement`, etc. and implement their functionality properly in libdom. This may involve adding `RadioNodeList` if we don't have that yet. It is likely to also involve needing to write tests which may or may not be unpleasant to do. 2. Repeat through that, ensuring that the NetSurf core functionality uses the new `libdom` interfaces properly. This will involve handling events, firing events, etc. as needed, as well as handling form data set generation (though that is mostly done, just may need tweaking wrt. which interfaces it uses). 3. Wire up JavaScript to everything now done. _Note_: 1. It's possible that we will need to work out a clean way for nodes in the DOM to listen to events. E.g. how does `dom_html_text_area_element` know when its `textContent` has changed? It needs to know, so that it can redo its `raw_value` and then fire appropriate events... 2. This will, if done properly, introduce a significant amount of intelligence into libdom about how values are represented, perhaps we will defer the validation stuff for now. 3. This will introduce more memory consumption for forms, though that is unlikely to be a large amount. 4. Select elements and option elements may become more complicated as a result, in order to cope with the reset algorithm. Also they will have to listen to their subtrees and update themselves if necessary. 5. Note that `form[name]` is a horrific horrific mess and may terrify the most hardened of cynics. Also note it's probably really heavily used. Oh Gods. This horror is implemented via a _past names map_ which is per-form. 6. Note that `form.elements` is not an `HTMLCollection` It's an `HTMLFormControlsCollection` which has a subtly different `.namedItem()` method. 7. Input elements can set a form owner directly using the `form` attribute (note not content attribute, dunno how that applies in `libdom`, perhaps we have to retain form owner somehow as an internal reference?) 8. Radio input elements introduce the concept of a radio button group which might need some fun calculation because it can happen in things without a `form_owner`. 9. File input elements might have a horrific value attribute involving nasty `C:\fakepath\` type bullshit. ARGH 10. Image input elements might be horrific, but we might be able to skip a lot of the "delay load event" type crap by simply ignoring it and going lalala 11. Things like input value attribute changes might need extra support in the layout engine to support things like caret movement. 12. File input elements have a `files` IDL attribute which returns the selected set of filenames. Do we manage that list on libdom for funsies? 13. Input and Change events are not fired by the DOM directly, but by the UA's input elements. This simplifies matters for us dramatically in the DOM but may mean extra spots of checking in the client. 14. Form association is a pig, reread carefully. Fortunately the `form` attribute appears to be readonly at the IDL level and as such we might get away with ignoring the point I made in _7_ above. 15. Select element's length is not readonly, and it adjusts the underlying DOM afaict. This horror, and associated gubbins is actually a reflection of the same APIs on `HTMLOptionsCollection`. Argh. 16. Progress, Meter, etc. are not actually form elements 17. Keyset doesn't need to be implemented 18. TextAreas are a pain, but don't look overcomplex beyond the above. 19. FieldSet will need to be properly implemented 20. The 'disabled' attribute of things will need careful consideration because it needs to flow through fieldsets etc. potentially. Perhaps fieldsets set the disabled on the children instead, but that's not clear for resetting. 21. The Legend element may need implementing so that `form_owner` can be presented to the UA. 22. The underlying DOM support for autofill might need implementing but frankly can wait, we're not going to deal with that shizzle for now. 23. I recommend we don't do any of the constraint validation for now, including minLength/maxLength stuff because it's a pig. History navigation scroll ========================= Daniel suggests: 1. Update `struct history_entry` to contain a `last_offset` as a float which is the proportion of the page scrolled (initialised to zero during `browser_window_history_add` 2. On navigation, if current is not NULL, store the `last_offset` into it (note in `browser_window_navigate`) 3. On page completion, if the current history entry's `last_offset` is non-zero then jump to that scroll location. Since we have fragment handling already (and navigating to a fragment will mean that the `last_offset` is zero) we will continue to function with fragments. Bug fixes ========= * Michael found and fixed a GTK core window issue where modifier keys were not getting passed with mouse clicks (release), only with press. This prevented Hotlist/Bookmarks entry editing. * Daniel fixed a set of leaks from the logging layer. Next time ========= We have chosen the next developer weekend to be . It shall be at