CouchDB - managing design documents?

Wed Sep 6 12:38:29 UTC 2017

Hi, All.

Thanks for your patience.  Monday, September 11th is the only day everyone
was available.  As everyone could make all three slots, I picked
16:00-17:00 CET.  I sent an invite to everyone who responded to the poll,
you can also see the meeting on the GPII Public Collaboration Calendar.

Cheers,

Tony

On Thu, Aug 31, 2017 at 8:40 PM, Harnum, Alan <aharnum at ocadu.ca> wrote:

> Thanks Tony – looking forward to talking about this.
>
>
>
> Gregor & I have been continuing work on an Infusion component for managing
> design documents (and a few other basic Couch config functions), since we
> have need of it for a current project: https://issues.fluidproject.
> org/browse/SJRK-6
>
>
>
> While I wouldn’t consider this close to final form, the example hopefully
> gets the intent across: https://github.com/waharnum/
> couch-config/blob/master/examples/js/couchConfigBasic.js
>
>
>
> I think JSON schema could be additionally powerful if we can find an
> approach to use them within the context of the validate_doc_update function
> of a design document. I think this would be possible using the require
> feature available in Couch: http://docs.couchdb.org/en/2.
> 0.0/query-server/javascript.html#require
>
>
>
> I could see a version of an Infusion-based CouchDB config component that
> did something like the following:
>
>
>
> 1)       Managed a set of JSON schema
>
> 2)       Loaded those schema to Couch
>
> 3)       Generated validate_doc_update functions appropriately based on
> those schema
>
>
>
> A common convention in Couch seems to be the use of a “type” field on
> documents that should be subject to validation, in order to access a
> particular set of validation rules. The “type” field in this scenario could
> be the JSON schema reference or similar.
>
>
>
> *From: *Tony Atkins <tony at raisingthefloor.org>
> *Date: *Wednesday, August 30, 2017 at 12:45 PM
> *To: *"Bates, Simon" <sbates at ocadu.ca>
> *Cc: *"Harnum, Alan" <aharnum at ocadu.ca>, Fluid Work <
> fluid-work at fluidproject.org>
>
> *Subject: *Re: CouchDB - managing design documents?
>
>
>
> Hi, All.
>
>
>
> Monday the 11th looks like the only day that works for everyone.  I am
> waiting to chat with Colin on his return before picking the time slot.
> Please hold 15:00-17:30 CET on that date for now, I will get back with the
> final time on the 4th.
>
>
>
> Cheers,
>
>
>
>
>
> Tony
>
>
>
> On Tue, Aug 29, 2017 at 11:14 AM, Tony Atkins <tony at raisingthefloor.org>
> wrote:
>
> Hi, Simon:
>
>
>
> That's a fair point, at a minimum we should be aware of the need and see
> what if any ground work we can take care of in working on managing design
> documents.
>
>
>
> As far as I know we do not currently have a general solution to managing
> data migrations.  In the rare cases when I've needed to restructure, I've
> used a bulk read, transform, and bulk update, but obviously that's only a
> very temporary solution.
>
>
>
> At the risk of entangling this conversation with another fairly big topic,
> one idea that comes to mind is using versioned JSON schemas and some
> convention of indicating what schema is used for a particular record (for
> example, a URI pointing to the schema).  That would let us:
>
>    1. Detect existing corrupt or incomplete records.
>    2. Limit a migration to a particular type and/or version of a record.
>    3. Verify that the results of the migration are themselves valid.
>    4. Determine which records have already been migrated, and which have
>    not.
>
> Anyway, we should bring up the topic in the architecture meeting and set
> up an initial meeting in the next week or so.  With that in mind, I have
> created a Doodle Poll:
>
>
>
> https://doodle.com/poll/snpxb4f496g289pk
>
>
>
> Cheers,
>
>
>
>
>
> Tony
>
>
>
> On Mon, Aug 28, 2017 at 5:31 PM, Bates, Simon <sbates at ocadu.ca> wrote:
>
> To broaden this topic slightly, I'm thinking that some form of data
> migration automation would also be important to have, in addition to
> updating design documents. For example, if a document structure is
> modified, we would want to have an automated way of updating existing data
> to the new structure.
>
> Do we have anything currently existing to manage data migration and
> document structure evolution? If not, I'm thinking it would be worth
> including the topic in discussions on managing design documents.
>
> Simon
>
> ________________________________________
> From: fluid-work <fluid-work-bounces at lists.idrc.ocad.ca> on behalf of
> Harnum, Alan <aharnum at ocadu.ca>
> Sent: August 28, 2017 11:19:59 AM
> To: Tony Atkins
> Cc: Fluid Work
> Subject: Re: CouchDB - managing design documents?
>
> Hi Tony,
>
> Thanks for the replies – we’ve also been looking at the various Couchapp
> implementations, but have been hesitant both for what you cite below (large
> additional stack of dependencies) and this part of the current CouchDB
> docs: http://docs.couchdb.org/en/2.1.0/ddocs/, which discourages the
> previously-promoted “CouchApps” paradigm that the various CouchApp tools
> are designed to use. My sense is that the various Couchapp projects are not
> under terribly active development in light of this change in CouchDB’s
> latest documentation.
>
> I’d also be an advocate of a Fluid component for capturing views and
> perhaps other configuration aspects of an Infusion application that intends
> to make use of CouchDB - I actually did some experimentation in this
> direction last Friday, which I’ve now cleaned up a bit and added some
> documentation: https://github.com/waharnum/couch-config
>
> In my ideal world I think I’d be able to write a CouchDB configuration for
> an application in pure Infusion rather than pushing JSON around with curl
> or other tools.
>
> From: Tony Atkins <tony at raisingthefloor.org>
> Date: Monday, August 28, 2017 at 7:32 AM
> To: "Harnum, Alan" <aharnum at ocadu.ca>
> Cc: Fluid Work <fluid-work at fluidproject.org>
> Subject: Re: CouchDB - managing design documents?
>
> Hi, Alan:
>
> Thanks again for starting yet another interesting and important
> discussion.  I've had to do a lot of this in my own work, and have
> alternated between:
>
>   1.  Storing the design documents as raw JSON (mainly for testing).  This
> is similar to the approach used in the example you shared.
>   2.  Writing design documents using couchapp<https://github.com/
> couchapp/couchapp>.
> The first approach seems easy enough, but is a bit tedious on a few fronts:
>
>   1.  It's difficult to write even moderately complex functions as escaped
> strings.
>   2.  Replacing an existing design document is a bit tedious, as it
> involves either looking up the current document's _rev or DELETEing and
> then POSTing the replacement.
>   3.  Although it's possible to require libraries from a design document<
> http://docs.couchdb.org/en/2.0.0/query-server/javascript.html#require>,
> this is pretty hard to manage using raw JSON files.
> Couchapp makes each of these easier:
>
>   1.  You just save your functions to a file, which you can run through
> our standard linting and more easily check for syntax errors.  You can even
> write unit tests against the functions you plan to use in your design
> documents.  This is especially helpful when developing reduce functions,
> which can be a bear to troubleshoot with CouchDB or PouchDB.
>   2.  Although it's trivial, couchapp makes it very easy to update design
> docs repeatedly.
>   3.  Couchapp makes it easier to bundle dependencies as part of your
> design document (basically by deploying them to a "lib" or similar
> directory within a design document's directory).
> That being said, couchapp is something I have only relied on reluctantly.
> I don't use it in tests, for example, as it seems a bit much to me to
> introduce a whole stack of dependencies (python, required libraries) just
> to solve what's a fairly simple use case.  Antranig and I have previously
> talked about building a Fluid component to assist with creating design
> documents, and it seems like a good time to broaden and refresh that
> discussion.
>
> Regardless of whomever else is interested, I would be very happy to work
> together with you (and I'm assuming Antranig) on a ticket outlining the use
> cases in more detail.  Depending on the scope of what we agree is needed, I
> might end up simply building it the next time I need to work on a batch of
> design documents.  I'd also be very happy to help review something someone
> else builds and test that in my own work.
>
> Cheers,
>
>
> Tony
>
> On Thu, Aug 24, 2017 at 4:16 PM, Harnum, Alan <aharnum at ocadu.ca<mailto:
> aharnum at ocadu.ca>> wrote:
> Hi all,
>
> We (Greg Moss & myself) are curious to know the opinions of others who've
> worked with it (especially in the context of Infusion / Kettle) about how
> to best manage design documents when using CouchDB as part of an
> application.
>
> Specifically, what approaches have worked for externalizing design
> document code, ensuring the database has the latest design document code,
> etc?
>
> I know one example of doing this is in gpii-oauth2-datastore, which is a
> JSON externalization of the views that is imported to the DB as part of
> application provisioning, described at: https://github.com/GPII/
> universal/tree/master/gpii/node_modules/gpii-oauth2/gpii-
> oauth2-datastore/dbViews
>
> I am wondering if people have seen or considered other approaches.
>
> My own feeling is that it would be useful to be able to model an expected
> CouchDB database design document as an Infusion component, with the
> component taking responsibility for ensuring a set of view functions are
> present in the specified DB, but I don't feel a great certainty about this.
>
> Curious as to the thoughts of others!
>
> Alan
>
> ALAN HARNUM
> SENIOR INCLUSIVE DEVELOPER
> INCLUSIVE DESIGN RESEARCH CENTRE, OCAD UNIVERSITY
>
> E aharnum at ocadu.ca<mailto://aharnum@ocadu.ca>
>
> OCAD UNIVERSITY
> 100 McCaul Street, Toronto, Canada, M5T 1W1
> www.ocadu.ca<http://ocadu.ca/>
>
> _______________________________________________________
> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca<mailto:
> fluid-work at lists.idrc.ocad.ca>
>
> To unsubscribe, change settings or access archives,
> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.inclusivedesign.ca/pipermail/fluid-work/attachments/20170906/231e0816/attachment.html>