Flexicon ========== A tool for individuals, business and crowds to organise and maintain a personal or community-owned compendium of compound commands and scripts, which can be standardised, shared, tracked and searched due to having a general compatible format and inbuilt tests. It is not tied to any particular operating system or software, it is a general tool to keep our working methods organised and to enable us to share and preserve them. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Index ----- A. Introduction Who uses Flexicons and why? How Tricks are used by the community B. Design Rationale Next steps Caveat Future Plans Project Motto C. Examples Use Case Sample Trick File: SvnShowRevisionNumber.tr Finding the right trick Checking if you can use a Flexicon D. Components List Trick Test Suite Environment/prerequisite checker Functionality checker E. Library program definition Basic Concept Commands User Interface Bookkeeping Preparation Shelf Management Testing books and tricks Producing books F. Data Fields Tricks Flexicon/Library =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= A. Introduction =============== Flexicon is tool to organise collections of compound commands (called tricks for short) that use one or more software applications to complete complex tasks. Each individual or community collects, writes and maintains their own Flexicons and individual tricks, and individual Flexicons are a named collection of offical tricks which are loaded by the main program, called 'Library' along with and personal tricks that the user can select on a case-by case basis. Tricks and Flexicons are freely interchangable between people, and ready to use (provided their integrated test passes). Flexicons and tricks can be integrity checked, and failing tricks can disabled until they have been repaired and are added back. Who uses Flexicons and why? ----------------------------- An individual may use Flexicon to keep a collection of their personal tricks, and tricks they obtain from other Phrasebooks or as individual items. A business can use Flexicon to organise tricks pertaining to particular IT tasks or positions, and to provide organisational memory, and a dictionary of processes used. A community can use Flexicon to pool effort and quality check tricks for use with their project or the product their community uses or produces. It is a communication tool & vault and specifically aims to provide a structure that prevents loss of information and obsolence via 'software rot' or lossage through time-induced obscurity, and enables a community to keep track of, and organise crowdsourced tricks. A project may use Flexicon to enable users to build a richer user interface than the project itself can offer. One such example is Apache Subversion which has a plethora of options that can be combined to do complex tasks, some of which are not immediately obvious even to expert users. This allows the dev team to add more bells and whistles to their project without the overhead of attempting to design how people will use their software easily, and allows users to standardise idioms they use and swap, or to create personal collections. A user community can produce a fully tested, validated Flexicon for every version of the projects software they use. Who better to design an user interface than the user community themselves? How Tricks are used by the community ------------------------------------ Tricks can be shared via the internet, and any trick from anywhere can be added to any Flexicon collection according to user-set conditions. Tricks may be publicly shared or kept private, they may be encrypted or even carry licenses. It's no different to what you find in the wild on the Internet, but with the added bonus of a set format and an attempt to ensure functional integrity. B. Design ========= Rationale --------- Flexicon provides a formal method of organising tricks into Flexicons which themselves can be collected into the library program that runs on the users' platform. It's a very general tool and not dependent on any particular usage, operating system or software. You can create, distribute and collect tricks for your windows system, your Linux/FreeBSD/Macintosh or whatever other system you're working on, provided the Flexicon software can run on your system. An individual trick may be a complex script which combines many different softwares, or it may be a simple one liner that just uses GNU tools for a particular task. Individual users can pick and mix tricks they like, and communities can choose to apply a voting process or another decision process as a quality control tool for their official compendiums. Most tricks come with a test suite that can be run individually or as part of an automated batch check to ensure that it works correctly in it's current setting. This ensures that the trick remains current despite a software upgrade or a project release, and that tricks can be selected correctly for particual software versions. Users can of course omit this for their personal tricks, or choose to download tricks without tests, at their own risk, which is not much different to the current situation of finding something useful on the internet. This may seem a bit complex, but it allows us to share our tricks and it is hoped that this gadget will save the Open Source community huge amounts of time since we no longer have to each knit our own tricks or rely on (and laboriously test) 'cooking recipes' we happen to find somewhere that may or may not work. Also, it will help to standardise what tricks we use. If one of us writes a trick to automagically (say) merge an svn trunk to an svn branch, everyone can partake if the creator shares this trick -- write once, use everywhere (relatively safely...!) is the motto! Next Steps ---------- I've posted a request to Apache Labs to attract contributers, hoping that this project will be promoted to Apache Incubator. Decisions need to be made what language and database to use, and the design idea needs a robust peer review. Many edges of the design are ragged, and there are a few parts where the description amounts to 'and here a miracle happens', in particular the current TrickServer concept. The actual size of the project is rather small --- so far the library, which is the actual heart of Flexicon has 27 user commands, most of which are classic queries that will be fun and easy to code. The programming language and other software to use needs to be decided upon. The requirements are: Small memory footprint and quick execution since were are writing something that actually is a meta shell, and as a second tier condition, ubiquity and as little user effort needed as is possbible for installation. Caveat ------ Thus spake Cassandra: "One user is difficult, two users gets tough, an unlimited amount of users with unlimited freedom to do whatever they want to do (some of which may be malicious!) is quite the tall order." Future Plans ------------ Once Flexicon is operational and past the initial test phrase, I would like to introduce the concept to the Subversion user community, in the hope that they indeed adopt Flexicon as a tool to build a user designed user interface to Apache Subversion and agree to stress test the concept, with a view to set up a voting process to allow the existing IRC svn-user community to collaborate on an official Apache Subversion Phrasebook. Project Motto ------------- Why have one or the other, when you can have one, the other, and both? C. Examples =========== Use Case -------- One typical application (and the orginal motivation) would be for the svn-user IRC community to collect and quality check compound commands that use svn in conjunction with with other software to produce a wrapper application for svn and other applications used in people's VCS processes. The other day on svn-dev mailing list, someone requested an svn feature to show the current revision number. Subversion didn't need a new feature, and whilst the solution _looks_ simple, it wasn't all that obvious either: svn info | sed -ne 's/Last Changed Revision: //p' We helped one user to a solution, but packaging the above trick into --svn-show-revision-number would help all users, and also a list of such tricks would inspire users to make full use of svn's capabilities they so far have been unaware of. The problem is a) how to share this efficiently b) how to keep those shared tricks up-to-date! If we can package the above as a trick together with a test, and present it in an accessible form, then all our users will have access to this knowledge and the trick can be automatically checked to see if it's still functional for every SVN release, and users can build up a compendium that educates, and works for, the entire community, (reasonably) realiably so. Ideally, the svn-user community on IRC will be able to use the Apache voting process and tools to produce and maintain proven Flexicons containing fully tested tricks which pertain to particular svn releases. If a trick fails it's tests, it's automatically removed from the official SVN Flexicon and placed back into the pending release Flexicon until it's fixed and voted back in again by the community members of svn-user. Sample Trick File: SvnShowRevisionNumber_1.tr ---------------------------------------------- \ID{1} \TrickName{SvnShowRevisionNumber} \Organisation{svn-user} \Flexicon{SVN-1.6-official} \Author{Gabriela Gibson} \Status{public} \Precis{Display current revision number of Subversion} \Description{Query the Subversion revision number using 'svn info'} \LongSwitch{--svn-show-revision-number} \Version{1.0} \Software{svn, grep} \Shell{bash} \Command{c=`$SVN info | grep "Last Changed Rev:"`; echo ${c##*: }} \Keywords{svn, revision} \TestExpectation{!integer} \InitialCreationDate{10 Dec 2013} \Last updated{10 Dec 2013} Finding the right trick ----------------------- If you perhaps noticed, the example trick file uses 'grep' and bash substitutions. What if you actually would prefer to use sh and sed? Tricks that have different execution methods but the same TrickName can be differentiated by searching their properties -- you look for a trick with the same name, that uses specific software: $ library find-software [location] --svn-show-revision-number sh,sed You can search with any field, or any combination thereof as the key. Example: Library also performs searches that help the user to produce a shelf with tricks/books that work, by just knowing their software requirements. Tricks may have many duplicates, for example: SvnShowRevisionNumber_1.tr (...) \Software{svn, grep} \Shell{bash} \Command{c=`$SVN info | grep "Last Changed Rev:"`; echo ${c##*: }} SvnShowRevisionNumber_2.tr (...) \Software{svn, sed} \Shell{sh} \Command{svn info | sed -ne 's/Last Changed Revision: //p'} Library will find and select the correct trick by looking at your software list, or check that you have the needed tools before it puts a trick on the shelf. So the ancient HP system that just has sh will have the same interface as someone's Macintosh, Windows or BSD system. Checking if you can use a Flexicon ------------------------------------ If you find a Flexicon that looks like it's perfect for you, you can run a check to see if you have all the required software on your system that the tricks in there need to use. $ library check-software FlexiconName will produce either a list of everything that is missing, or, inform you that it's ready to be used. About TrickServer ----------------- TrickServer is a small webhosted database that registers tricks and serves unique IDs to new trick authors. TrickServer can be searched by [TrickName] [keywords] [software] and will return ID's of matching tricks. If it cannot find any tricks that match, it will inform you that this trick is missing, or if it only finds partial matches, it will return all the ID's that match, which you then can google for and see if you can find it on some mailing list or someone's blog perhaps. Note: I'm not sure about this design, it looks like it could be vandalised. Maybe using an email list and utilising the unique SMPT id might just about work, but I think this is beyond my current expertise and I would like to invite interested readers to propose a better concept here. D. Components ============= Trick -- a plain text file in LaTeX macro format. Flexicon -- a plain text file in a dedicated directory which contains information about the Flexicon and a list of tricks which reside in the same directory. Library -- command line program that organises them and creates new Flexicons. TrickServer -- central server that assigns unique ID and collects (some) information about tricks. It also facilitates a voting process in order to purge obsolete tricks. Trick Test Suite: ----------------- Tests are run before its added to a Library on a system, or as a requested check, for example, after software upgrades. Individuals may decide to ignore failing tests, but communities may have their own procedures to validate tricks. Environment/prerequisite checker -------------------------------- Checks that the test matches the current environment -- if you need zsh and svn, this checks for the availability those softwares and ensures that the versions thereof are in the range as defined by the trick author(s). Similiar in concept to make's configure. Functionality checker --------------------- Runs the trick's test (if it exists) that checks whether the trick or Flexicon has the correct output on your system. E. Library program definition ============================= Basic concept: Library is the user interface that manages tricks and books and runs the commands. Library has general and optional named shelves on which it stores collection of tricks obtained from Flexicons and individual tricks. A shelf can be emptied, filled, combined and packaged into a new book, or dumped out to an alias file. Library will check the users selection to ensure that every trick on the shelf works, and mark broken tricks/books as 'compromised' until their tests no longer fails. Commands ======== There are currently 27 command designs for the library program: User Interface: library Bookkeeping: makeAliasFile, make-trick-form Preparation: check-trick, check-book, check-software, find-software, gather-software, diff-software, select-trick, purge-trick Shelf management: remove-book, remove-shelf, remove-trick, add-book, add-trick, use-book, use-trick, show-shelf, trick-show-[datafields], book-show-[datafields], move-trick, copy-trick Testing books and tricks: validate-trick, validate-shelf, validate-book, validate-show-failures Producing books: write-book, gift-wrap User interface --------------- $ library [TrickName|switch] run this trick on system. If library cannot parse your input, the error message is 'Ook?' followed by an attempt to explain what went wrong. library itself can be part of a piped compound command under UNIX -- tricks can contain tricks that contain tricks which contain tricks(...) Bookkeeping: ------------ List: makeAliasFile, make-trick-form $ makeAliasFile [shelf|{TrickName1, TrickName2...}] [format] [filename] Dump requested library contents into an alias file(s): TrickName=command for unix, not sure what windows needs. $ make-trick-form [TrickFileName, ID] Create TrickFileName_ID.tr containing skeleton of datafields. Preparation: ------------ List: check-trick, check-book, check-software, find-software, gather-software, diff-software, select-trick, purge-trick $ check-trick [optional: --mandatory] reports any missing entries in a trick file. If mandatory is set only missing mandatory entries are shown. $ check-book [optional: --mandatory] [Flexicon] As check-trick, but it check all the tricks of the Flexicon. $ check-software [TrickName|Flexicon|shelf|all] Create list of software required and runs 'configure' to check if the user's system has the required software, report back if 'foo' needs to be installed, disable non-functional tricks, enable all tricks that work. $ find-software [switch|TrickName|keywords] [software] [location] Search location for tricks that conform to software requirements. Do not show duplicates if one trick already matches the requirements. Report tricks/books that are not yet on the shelf, and have software requirements $ gather-software [switch|TrickName|keywords] [software] [location] [FlexiconName] Search location for tricks that conform to software requirements. $ diff-software [switch|TrickName] [location] Show a numbered list along with the software requirement and command differences of the given TrickName. $ select-trick [number|ID] Sometimes two or more tricks do the same thing in a different way. This command allows you to select the one your prefer. $ purge-trick [number|ID] [successor ID | alternative ID] Schedules an obsolete trick for a deletion vote at the TrickServer. Note that the ID is never reused, but marked obsolete. A duplicated trick that uses a different command is not obsolete, nor is a broken trick that could be repaired. However an identical trick that has two IDs is indeed obsolete -- there can only be one! Shelf management: ----------------- List: remove-book, remove-shelf, remove-trick, add-book, add-trick, use-book, use-trick, show-shelf, trick-show-[datafields], book-show-[datafields], move-trick, copy-trick Tricks and Flexicons are never actually deleted, just removed from the library's data collection. If you want to delete them, you need to delete the actual file(s) themselves. $ remove-book [FlexiconName] removes everything from FlexiconName from all shelves. $ remove-shelf [shelf] empty that shelf. $ remove-trick [TrickName|switch] Remove [TrickName|switch] from the shelves. $ add-book [no-test] [shelf] [FlexiconName] If no shelf is given, a new shelf bearing FlexiconName is created. $ add-trick [no-test] [shelf] [TrickName] If no shelf is given, it will be added to 'general'. If no-test is set, the addition will not be tested but marked 'compromised'. $ use-book [FlexiconName] Empty all shelves, load FlexiconName $ use-trick [TrickName] Empty all shelves, use TrickName instead. $ show-shelf [shelf|all] [switches|TrickName] show what tricks are on the shelf and the original FlexiconName if applicable. $ trick-show-[datafields] [switch|TrickName|keywords] where [datafields] are any data fields a Trick contains. Example: trick-show-precis-command $ book-show-[datafields] [Flexicon|shelf|keywords] where [datafields] are any data fields a Trick contains. Example: book-show-author-status $ move-trick [TrickName] [shelf] move trick to a shelf. $ copy-trick [TrickName] [shelf] copy trick to a shelf. Testing books and tricks ------------------------ List: validate-trick, validate-shelf, validate-book, validate-show-failures $ validate-trick [Name] $ validate-shelf [shelf] $ validate-book [Flexicon] run tests on the above. Tricks marked 'Compromised' will be disabled. Show report of disabled items. $ validate-show-failures display complete report of failed tests. Producing books: ---------------- List: write-book, gift-wrap $ write-book [shelf] write new book from shelf contents. $ gift-wrap [FlexiconName] add author's digital signature and make md5sum (I assume you can securely add the actual md5sum into the signature somehow, so malicious modification is not easily possible despite it being plain text) F. Data Fields ============== A trick is a plain text file using the LaTeX macro style format, ie, \fieldName{} The following fields are defined. * denotes an mandatory entry to ensure overall consistency + denotes user/community set mandatory entry - denotes optional entry that users/communities can set. & encrypted mandatory field (probably a key for a server?) % encrypted optional field (probably a key for a server?) # constant mandatory entry R range (min, max, numerical, in []) Absence of a non-mandatory field is interpreted as 'none'. Tricks ------ + Flexicon Name of Flexicon this belongs to + Organisation Name of community/company # Author The original designer # ID unique numerical ID % AuthorEmail Email of original designer + Maintainer Current maintainer % MaintainerEmail Email of maintainer - MailingList mailing list for this trick - Contributers people who made/fixed this - PublicKey To sign the trick - md5sum Enable safety check of trick * Status public, restricted, personal + License Licence type + Copyright some tricks are propietary + Encryption whether the trick is secured - Reviewers last person/group to review this # TrickName One word alias name * Precis Short description (one line) * Description Long description * LongSwitch official Long switch + ShortSwitch official short switch - UserLongSwitch user set alias for long switch - UserShortSwitch user set alias for short switch * Version tricks have versions + OperatingSystem, R what OS to use (absence = any) + ExportVariables shell scripts may require env vars set + ExportVariablesDescription description of req. env vars * Software list of req. SW and [version ranges] * Shell What shell to use * Command script or command of the trick * Keywords a list of tags eg, bash, svn, emacs + TestShell What driver to use to run the test + TestEnvironment environment script for the test + TestInput data for the test + TestCode code for the test + TestExpectation expected output + TestResult expected result + FunctionalityStatus result of last test run - Notes Extra notes + Bugs Bugs and such + Alternatives Similiar tricks (IDs thereof) + Ancestor Trick that was used to make this + Successor Newer version of trick (ID thereof) * InitialCreationDate When was it made? * LastUpdated When was it last fixed? + WebAddress Canonical source of this trick + UserDefinedFields Some empty spots for future use + Compromised true if a test failed + Server Origin Flexicon/Library ------------------ # Real ID Unique ID * Working ID for inofficial/personal copies # FlexiconName Name - Public Key To sign the Flexicon - md5sum Enable safety check of Flexicon + OperatingSystem, R what OS to use (absence = any) * Software list of req. SW and version ranges * Shell,R What shell to use * Precis Short description (one line) * Description Long description + Tricklist list of tricks added * InitialCreationDate When was it made? * LastUpdated When was it last updated? + WebAddress Canonical source of this Flexicon + Alternatives Similiar Flexicons (IDs thereof) + Ancestor Original Flexicon or None + Successor Newer version of Flexicon (ID thereof) * Keywords a list of tags eg, bash, svn, emacs + Organisation Name of community/company # Author The original designer % AuthorEmail Email of original designer + Maintainer Current maintainer % MaintainerEmail Email of maintainer - Mailing list mailing list for this trick - Contributers people who made/fixed this * status public, restricted, personal + License Licence type + Copyright some Flexicons are propietary + Encryption whether the trick is secured - Reviewers last person/group to review this + Compromised true if a test failed