summaryrefslogtreecommitdiff
path: root/developer-weekend/sep-2017.mdwn
blob: f7305facde916876d9023989780e2dafeae3f164 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
[[!meta title="Developer Weekend (September 2017)"]]
[[!meta author="Daniel Silverstone"]]
[[!meta date="2017-09-00 17:30:00"]]

[[!toc]]

Attendees
=========

* Michael Drake
* Vincent Sanders
* Daniel Silverstone

Apologies
=========

* John-Mark Bell
* Chris Young

Topics
======

* Logging merge for main NetSurf
    * Daniel added verbose and normal logging options
    * Vince added documentation for nslog in netsurf
    * Vince fixed a whole pile of horrible compile problems with nslog on atari etc.
    * Daniel fixed the test suite to work with the new logging options
* Triage of bug tracker to generate further topics for Saturday
    * Michael found a couple of 10+ year old feature requests assigned to him, and decided
      it was time to implement them, so he got started on:
        * [`#1949` Search option for hotlist](http://bugs.netsurf-browser.org/mantis/view.php?id=1949)
        * [`#1954` No way to search global history](http://bugs.netsurf-browser.org/mantis/view.php?id=1954)
    * Daniel fixed [`#2549`](http://bugs.netsurf-browser.org/mantis/view.php?id=2549)
    * Daniel reviewed [`#2537`](http://bugs.netsurf-browser.org/mantis/view.php?id=2537)
      and decided that it wasn't worth trying to fix now.  It's reported against
      3.6, the reported issue is fixed, but the underlying inability to interact
      with buttons is not.
    * Daniel and vince reviewed [`#2555`](http://bugs.netsurf-browser.org/mantis/view.php?id=2555)
      and decided François had already fixed it.
    * Daniel reviewed [`#2544`](http://bugs.netsurf-browser.org/mantis/view.php?id=2544)
      and decided it had always worked, closed it can't reproduce.
    * Daniel tried to reproduce [`#2532`](http://bugs.netsurf-browser.org/mantis/view.php?id=2532)
      but failed, so asked the submitter to try again.
    * Daniel looked at:
        * [`#2464`](http://bugs.netsurf-browser.org/mantis/view.php?id=2464)
        * [`#2482`](http://bugs.netsurf-browser.org/mantis/view.php?id=2482)
        * [`#2417`](http://bugs.netsurf-browser.org/mantis/view.php?id=2417)
      and closed them unable-to-reproduce.
    * Daniel reviewed [`#526`](http://bugs.netsurf-browser.org/mantis/view.php?id=526)
      and assigned it to Michael
    * Vince reviewed [`#2550`](http://bugs.netsurf-browser.org/mantis/view.php?id=2550)
      and closed it unable-to-reproduce.
    * Daniel and Vince worked through the following, closing most of them...
        * [`#1885`](http://bugs.netsurf-browser.org/mantis/view.php?id=1885)
		* [`#1886`](http://bugs.netsurf-browser.org/mantis/view.php?id=1886)
		* [`#1889`](http://bugs.netsurf-browser.org/mantis/view.php?id=1889)
		* [`#420`](http://bugs.netsurf-browser.org/mantis/view.php?id=420)
		* [`#434`](http://bugs.netsurf-browser.org/mantis/view.php?id=434)
	    * [`#518`](http://bugs.netsurf-browser.org/mantis/view.php?id=518) (Actually we resolved this one)
		* [`#469`](http://bugs.netsurf-browser.org/mantis/view.php?id=469) (Actually we resolved this one)
		* [`#520`](http://bugs.netsurf-browser.org/mantis/view.php?id=520) (Actually we resolved this one)
		* [`#522`](http://bugs.netsurf-browser.org/mantis/view.php?id=522)
		* [`#505`](http://bugs.netsurf-browser.org/mantis/view.php?id=505)
	    * [`#1916`](http://bugs.netsurf-browser.org/mantis/view.php?id=1916) (We won'tfix'd this)
        * [`#1907`](http://bugs.netsurf-browser.org/mantis/view.php?id=1907) (We trolled jmb)
		* [`#1879`](http://bugs.netsurf-browser.org/mantis/view.php?id=1879) (Resolved)
        * [`#485`](http://bugs.netsurf-browser.org/mantis/view.php?id=485)

* What frontends do we demote?
    * See below

* Administrivia - Phoenix's disks are tired (even SMART said so)
    * We'd like to know where the society bank account stands, for general
      resources.
    * We'd like to buy a pair of WD Gold 1TB drives (rough cost 140 UKP total)
      so that Phoenix isn't running on pre-fail age drives.
    * We investigated cloud/hosted servers, sadly it's super-expensive compared
      to some new disks for the free hosting we have.
    * Decision was: Michael, Vince, and Daniel will share the cost if the
      society lacks the funds.  Vince will buy drives and begin the replacement
      process.  Ideally ASAP.

* Discussion around forms, removing gadgets, etc.
    * See below

* Auto-scroll to last position during history navigation
    * See below

* Breaking changes and 4.x
    * We have defined 'major' changes as anything large (e.g. new layout would
      be major)
    * Also 'major' changes include on-disk format changes.  This means that
      we'll forward-migrate where possible, but make no guarantees about going
      backwards.  Migration will be explicit and cancellable on startup.
    * Content migrations must be such that if you run an old version, it won't
      pick up old content by default.

* What's left to go in to 3.7
    * Michael's searching for hotlist (treeview search)
    * Vince's fixes for bitmaps in local history
    * History navigation keeping scroll offsets
    * Logging rework fallout
    * General memory leak fixes where possible
    * Aiming for 3.7 release handle crankage around Saturday September 23rd.

* URLdb/Cookies
    * URLdb is a mess.  We knew that, there's nothing new there.
    * We'd like to replace it, in its entirety, including on-disk formats.
    * It should be split into a proper cookie jar, and a metadata store keyed
      by URL.
    * For the cookie jar, there is an RFC for cookie handling, lots of useful
      information and links on
      <https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies>
    * For the "URL database" we need to store arbitrary and changeable sets
      of key-to-value mappings, some of which need to persist, some of which
      are to be RAM-only.
    * If we add scope to the 'key' such that we can have localstorage as a
      scope then we can put local storage <https://www.w3.org/TR/webstorage/>
      into the same DB.
    * Potential implementation strategy:
        * Use a journalling approach (where the entries are
          create/change and delete commands) storing the journal as the URLdb
		  replacement.
        * The primary key into the DB is based on `nsurl`.
        * We will reuse the `resource:` scheme for "internal" storage.
          (e.g. session state, disk cache indices etc)
        * The "value set" is a bag of triples. (`scope`, `key`, `value`) where
          the scope is one of an enumeration (e.g. `localStorage` or `urldb`)
          the key is a simple utf-8 string, and the value is an arbitrary set
          of bytes (up to 4GB). (thus is stored as bytes+u32_length).
        * We need the in-memory representation to support the following
          efficiently:
            1. Counting of key,value pairs in a given scope for a given url
            2. Indexing a given (url,scope) pair by integer stably
            3. Indexing a given (url,scope) pair by key name
            4. Construction of journal entries (on changes)
            5. Completion of url strings
            6. Distinguish between content which can be persisted and that
               which should not.
    * The "clean" writeout should ensure that everything necessary for the
      session restoration gets written out first, and then a marker is put down
      which the readback code can use to kick off an async load of the rest of
      the data.
    * File format should be binary, and be robust (length, kind, checksum,
      payload).
    * Plausible things to layer on this generic storage are:
        * URLdb replacement
        * Session management
        * LocalStorage
        * Cookie jar
        * File-based cache backend (indices)
        * User options.
    * Multiplicity (no single core)
    * Proper testing for this.

* The "What's Next?" of Javascript
    * We do all the other stuff first, since it all has JS implications
    * We need to update the IDLs and include more parts (e.g. LocalStorage,
      CSSOM, etc)
    * nsgenbind - should it do more automatic generation? If so how?
    * Fix nsgenbind's `#file` `#line` stuff so that debugging works better.
    * Fill out more of the bindings.
    * So much of libdom needs enhancing too.



Frontends
=========

Amiga
-----

Chris Young still maintaining it.  Do we want to add OS3 to the CI?

> Keep '1st class'

Atari
-----

No maintainer, toolchain is awkward, blobby, unpleasant for CI.  Needs a
scrub and refactor.  We're not sure our builds are valid for Coldfire.

> Demote '2nd class' (no CI builds) after 3.7

BeOS
----

Intermittent maintenance.  Haiku builder is a pain in the arse, is the only
blocker for Java 8.

> Keep `1st class` for now, consult with Pulkomandy and mmu_man about JDK-8

Cocoa
-----

Currently 2nd class, Libraries only.  Sven came back, but made a messy branch
which was too hard to review/merge.  Then he vanished once more.  Is also a JDK
8 limiting blocker, as it is 2nd class, we'd just drop from CI entirely if it
was the only blocker.

> Keep `2nd class` for now.

Framebuffer
-----------

Currently 1st class, boring, needs keyboard shortcuts etc.  Hotlist support,
blahblahblah but frankly "just works™" for the most part (Raw linux framebuffer
doesn't work, but that's `libnsfb` really)

> Keep `1st class` for now.

GTK
---

Currently 1st class, basically works, lacks keyboard shortcuts and polish in
general.

> Keep `1st class` for now.

Monkey
------

Currently 1st class, doesn't tend to cause problems, needs further work for
a test driver etc.  No proper maintainer right now.

> Keep `1st class` for now.

RISC OS
-------

No maintainer, none of the core devs actually understand it any more, it's a
pain in the arse.  Sadly most of our feedback comes from RISC OS users.  For
now we do a best-effort on keeping it first class.

> Keep `1st class` for now.

Windows
-------

Minimally usable, but there are problems with XP (though do we care?) some
issues around certificate bundles, storing of userdata, etc.  Needs better
configuration, translations, etc.

> Keep `1st class` for now.

Forms and gadgets
=================

The issues as we see them are that basically it's impossible to make small
fixes because whenever something is changed, it goes horribly wrong somewhere
else.  Usually manifesting in double-frees or similar given the particularly
baroque ownership semantics (or lack thereof).

Rather than hacking it all out and starting from scratch, we proposed:

1. Making sure the gadgets update the DOM
    * Gadget types: `HIDDEN`, `TEXTBOX`, `RADIO`, `CHECKBOX`, `SELECT`,
      `TEXTAREA`, `IMAGE`, `PASSWORD`, `SUBMIT`, `RESET`, `FILE`, `BUTTON`
    * It's likely `HIDDEN` can go away as soon as 2 is done.
    * Validate that all these input kinds are supported in the libdom binding
        * **TODO** Need to work out how to handle the reset event
2. Making the form submission read from the DOM
    * Turns out Daniel wrote that in 2014. Oops. Nothing to do here.
3. Make the DOM trigger the submission (may need onclick or other event stuff)
    * **TODO** First up read
      <https://www.w3.org/TR/html5/forms.html#concept-fs-action> and implement
      the form action, method, etc attributes onto the input element etc.
      This may need libdom tweaks for the attributes and the novalidate stuff.
    * **TODO** Rework the libdom submit stuff to use an internal event name
      because the `submit` event is for non-`form.submit()` stuff:
      <https://www.w3.org/TR/html5/forms.html#concept-form-submit>
    * **TODO** Now determine how to implement the above bits neatly so that
      we trigger submission properly.
4. Make any reader of the gadgets use the DOM
5. Work out how to remove more of this...

**Sadly it turned out not to be that simple**

New approach
------------

1. Using <https://www.w3.org/TR/html5/forms.html> go through each of
   `HTMLFormElement`, `HTMLInputElement`, `HTMLButtonElement`,
   `HTMLTextAreaElement`, etc. and implement their functionality properly in
   libdom.  This may involve adding `RadioNodeList` if we don't have that yet.
   It is likely to also involve needing to write tests which may or may not be
   unpleasant to do.
2. Repeat through that, ensuring that the NetSurf core functionality uses the
   new `libdom` interfaces properly.  This will involve handling events, firing
   events, etc. as needed, as well as handling form data set generation (though
   that is mostly done, just may need tweaking wrt. which interfaces it uses).
3. Wire up JavaScript to everything now done.

_Note_:

1. It's possible that we will need to work out a clean way for nodes in the
   DOM to listen to events.  E.g. how does `dom_html_text_area_element` know
   when its `textContent` has changed?  It needs to know, so that it can redo
   its `raw_value` and then fire appropriate events...
2. This will, if done properly, introduce a significant amount of intelligence
   into libdom about how values are represented, perhaps we will defer the
   validation stuff for now.
3. This will introduce more memory consumption for forms, though that is
   unlikely to be a large amount.
4. Select elements and option elements may become more complicated as a result,
   in order to cope with the reset algorithm.  Also they will have to listen
   to their subtrees and update themselves if necessary.
5. Note that `form[name]` is a horrific horrific mess and may terrify the most
   hardened of cynics.  Also note it's probably really heavily used. Oh Gods.
   This horror is implemented via a _past names map_ which is per-form.
6. Note that `form.elements` is not an `HTMLCollection` It's an
   `HTMLFormControlsCollection` which has a subtly different `.namedItem()`
   method.
7. Input elements can set a form owner directly using the `form` attribute
   (note not content attribute, dunno how that applies in `libdom`, perhaps
   we have to retain form owner somehow as an internal reference?)
8. Radio input elements introduce the concept of a radio button group which
   might need some fun calculation because it can happen in things without
   a `form_owner`.
9. File input elements might have a horrific value attribute involving
   nasty `C:\fakepath\` type bullshit.  ARGH
10. Image input elements might be horrific, but we might be able to skip a lot
    of the "delay load event" type crap by simply ignoring it and going lalala
11. Things like input value attribute changes might need extra support in the
    layout engine to support things like caret movement.
12. File input elements have a `files` IDL attribute which returns the selected
    set of filenames.  Do we manage that list on libdom for funsies?
13. Input and Change events are not fired by the DOM directly, but by the UA's
    input elements.  This simplifies matters for us dramatically in the DOM but
    may mean extra spots of checking in the client.
14. Form association is a pig, reread
    <https://www.w3.org/TR/html5/forms.html#attr-fae-form> carefully.
    Fortunately the `form` attribute appears to be readonly at the IDL level
    and as such we might get away with ignoring the point I made in _7_ above.
15. Select element's length is not readonly, and it adjusts the underlying DOM
    afaict.  This horror, and associated gubbins is actually a reflection of
    the same APIs on `HTMLOptionsCollection`.  Argh.
16. Progress, Meter, etc. are not actually form elements
17. Keyset doesn't need to be implemented
18. TextAreas are a pain, but don't look overcomplex beyond the above.
19. FieldSet will need to be properly implemented
20. The 'disabled' attribute of things will need careful consideration because
    it needs to flow through fieldsets etc. potentially.  Perhaps fieldsets set
    the disabled on the children instead, but that's not clear for resetting.
21. The Legend element may need implementing so that `form_owner` can be
    presented to the UA.
22. The underlying DOM support for autofill might need implementing but frankly
    can wait, we're not going to deal with that shizzle for now.
23. I recommend we don't do any of the constraint validation for now, including
    minLength/maxLength stuff because it's a pig.


History navigation scroll
=========================

Daniel suggests:

1. Update `struct history_entry` to contain a `last_offset` as a float
   which is the proportion of the page scrolled (initialised to zero during
   `browser_window_history_add`
2. On navigation, if current is not NULL, store the `last_offset` into it
   (note in `browser_window_navigate`)
3. On page completion, if the current history entry's `last_offset` is non-zero
   then jump to that scroll location.

Since we have fragment handling already (and navigating to a fragment will mean
that the `last_offset` is zero) we will continue to function with fragments.

Bug fixes
=========

* Michael found and fixed a GTK core window issue where modifier keys were not
  getting passed with mouse clicks (release), only with press.  This prevented
  Hotlist/Bookmarks entry editing.
* Daniel fixed a set of leaks from the logging layer.

Next time
=========

We have chosen the next developer weekend to be .
It shall be at