diff options
Diffstat (limited to 'tdeio/DESIGN')
| -rw-r--r-- | tdeio/DESIGN | 272 |
1 files changed, 272 insertions, 0 deletions
diff --git a/tdeio/DESIGN b/tdeio/DESIGN new file mode 100644 index 000000000..22307c5c0 --- /dev/null +++ b/tdeio/DESIGN @@ -0,0 +1,272 @@ +DESIGN: +======= + +libtdeio uses tdeioslaves (separate processes) that handle a given protocol. +Launching those slaves is taken care of by the tdeinit/tdelauncher tandem, +which are notified by DCOP. + +Connection is the most low-level class, the one that encapsulates the pipe. + +SlaveInterface is the main class for transferring anything to the slave +and Slave, which inherits SlaveInterface, is the sub class that Job should handle. + +A slave inherits SlaveBase, which is the other half of SlaveInterface. + +The scheduling is supposed to be on a two level basis. One is in the daemon +and one is in the application. The daemon one (as opposite to the holy one? :) +will determine how many slaves are ok for this app to be opened and it will +also assign tasks to actually existing slaves. +The application will still have some kind of a scheduler, but it should be +a lot simpler as it doesn't have to decide anything besides which +task goes to which pool of slaves (related to the protocol/host/user/port) +and move tasks around. +Currently a design study to name it cool is in scheduler.cpp but in the +application side. This is just to test other things like recursive jobs +and signals/slots within SlaveInterface. If someone feels brave, the scheduler +is yours! +On a second thought: at the daemon side there is no real scheduler, but a +pool of slaves. So what we need is some kind of load calculation of the +scheduler in the application and load balancing in the daemon. + +A third thought: Maybe the daemon can just take care of a number of 'unused' +slaves. When an application needs a slave, it can request it from the daemon. +The application will get one, either from the pool of unused slaves, +or a new one will be created. This keeps things simple at the daemon level. +It is up to the application to give the slaves back to the daemon. +The scheduler in the application must take care not to request too many +slaves and could implement priorities. + +Thought on usage: +* Typically a single slave-type is used exclusively in one application. E.g. +http slaves are used in a web-browser. POP3 slaves used in a mail program. + +* Sometimes a single program can have multiple roles. E.g. konqueror is +both a web-browser and a file-manager. As a web-browser it primarily uses +http-slaves as a file-manager file-slaves. + +* Selecting a link in konqueror: konqueror does a partial download of +the file to check the mimetype (right??) then the application is +started which downloads the complete file. In this case it should +be able to pass the slave which does the partial download from konqueror +to the application where it can do the complete download. + +Do we need to have a hard limit on the number of slaves/host? +It seems so, because some protocols are about to fail if you +have two slaves running in parralel (e.g. POP3) +This has to be implemented in the daemon because only at daemon +level all the slaves are known. As a consequence slaves must +be returned to the daemon before connecting to another host. +(Returning the slaves back to the daemon after every job is not +strictly needed and only causes extra overhead) + +Instead of actually returning the slave to the daemon, it could +be enough to ask 'recycling permission' from the daemon: the +application asks the daemon whether it is ok to use a slave for +another host. The daemon can then update its administration of +which slave is connected to which host. + +The above does of course not apply to hostless protocols (like file). +(They will never change host). + +Apart from a 'hard limit' on the number of slaves/host we can have +a 'soft limit'. E.g. upon connection to a HTTP 1.1 server, the web- +server tells the slave the number of parallel connections allowed. +THe simplest solution seems to be to treat 'soft limits' the same +as 'hard limits'. This means that the slave has to communicate the +'soft limit' to the daemon. + +Jobs using multiple slaves. + +If a job needs multiple slaves in parallel (e.g. copying a file from +a web-server to a ftp-server or browsing a tar-file on a ftp-site) +we must make sure to request the daemon for all slaves together since +otherwise there is a risk of deadlock. + +(If two applications both need a 'pop3' and a 'ftp' slave for a single +job and only a single slave/host is allowed for pop3 and ftp, we must +prevent giving the single pop3 slave to application #1 and the single +ftp slave to application #2. Both applications will then wait till the +end of times till they get the other slave so that they can start the +job. (This is a quite unlikely situation, but nevertheless possible)) + + +File Operations: +listRecursive is implemented as listDir and finding out if in the result + is a directory. If there is, another listDir job is issued. As listDir + is a readonly operation it fails when a directory isn't readable + .. but the main job goes on and discards the error, because +bIgnoreSubJobsError is true, which is what we want (David) + +del is implemented as listRecursive, removing all files and removing all + empty directories. This basically means if one directory isn't readable + we don't remove it as listRecursive didn't find it. But the del will later + on try to remove it's parent directory and fail. But there are cases when + it would be possible to delete the dir in chmod the dir before. On the + other hand del("/") shouldn't list the whole file system and remove all + user owned files just to find out it can't remove everything else (this + basically means we have to take care of things we can remove before we try) + + ... Well, rm -rf / refuses to do anything, so we should just do the same: + use a listRecursive with bIgnoreSubJobsError = false. If anything can't + be removed, we just abort. (David) + + ... My concern was more that the fact we can list / doesn't mean we can + remove it. So we shouldn't remove everything we could list without checking + we can. But then the question arises how do we check whether we can remove it? + (Stephan) + + ... I was wrong, rm -rf /, even as a user, lists everything and removes + everything it can (don't try this at home!). I don't think we can do + better, unless we add a protocol-dependent "canDelete(path)", which is + _really_ not easy to implement, whatever protocol. (David) + + +Lib docu +======== + +mkdir: ... + +rmdir: ... + +chmod: ... + +special: ... + +stat: ... + +get is implemented as TransferJob. Clients get 'data' signals with the data. +A data block of zero size indicates end of data (EOD) + +put is implemented as TransferJob. Clients have to connect to the +'dataReq' signal. The slave will call you when it needs your data. + +mimetype: ... + +file_copy: copies a single file, either using CMD_COPY if the slave + supports that or get & put otherwise. + +file_move: moves a single file, either using CMD_RENAME if the slave + supports that, CMD_COPY + del otherwise, or eventually + get & put & del. + +file_delete: delete a single file. + +copy: copies a file or directory, recursively if the latter + +move: moves a file or directory, recursively if the latter + +del: deletes a file or directory, recursively if the latter + +PROGRESS DISPLAYING : +===================== +Taj brought up the idea of deligating all progress informations to an extern +GUI daemon which could be provided in several implementations - examples +are popup dialogs (most are annoyed by them, like me :) or a kicker applet +or something completely different. This would also remove the dependency on +libtdeui (I hope). +Conclusion: tdeio_uiserver is this single GUI daemon, but the dependency on +libtdeui couldn't be removed (for many reasons, including Job::showErrorDialog()) + +A. progress handling +--------------------- +There will be two ways how the application can display progress : + +1. regular apps will use NetAccess for all kio operations and will not care + about progress handling : + - NetAccess creates Job + - NetAccess creates JobObserver that will connect to the Job's signals and + pass them via dcop to the running GUI Progress Server + +2. apps that want to do some handling with progress dialogs like Caitoo or + KMail : + - app creates Job + - app creates a progress dialog : this should be a ProgressBase descendant + e.g. StatusProgress or custom progress dialog + - app calls progress->setJob( job ) in order to connect job's signals with + progress dialog slots + +B. customized progress dialogs +------------------------------- + This will be similar to what we had before. + + - ProgressBase class that all other dialogs will inherit. + will contain an initialization method setJob( TDEIO::Job*) for apps of the + second class (see A.2 above), that will connect job's signals to dialog's + slots + + - DefaultProgress ( former KIOSimpleProgressDialog ) that will be used for + regular progress dialogs created by GUI Progress Server + + - StatusProgress ( former KIOLittleProgressDialog ) that can be used for + embedding in status bar + +C. GUI Progress Server +----------------------- + This is a special progress server. + - createProgress() will either create a DefaultProgress dialog or add new entry + in a ListProgress ( an all-jobs-in-one progress dialog ) + - after receiving signals from the JobObserver via DCOP it will call + appropriate method of progress dialog ( either in DefaultProgress or ListProgress ) + - ListProgres can be a Caitoo style dialog, kicker applet or both in one. + +D. Some notes +-------------- + 1. most of the apps will not care at all about the progress display + 2. user will be able to choose whether he wants to see separate progress + dialogs or all-in-one ListProgress dialog + 3. developers can create their custom progress dialogs that inherit + ProgressBase and do any manipulation with a dialog if they use a second + approach ( see A.2 above ) + + +Streaming +--------- + + 1. We currently support a streaming "GET": e.g. file:/tmp/test.gz#gzip:/ + works. The following should also work: file:/tmp/test.gz.gz#gzip:/#gzip:/ + The current approach makes a TrasnferJob for gzip:/ and then adds a + subjob for "file:/tmp/test.gz.gz#gzip:/" which itself adds a subjob + for "file:/tmp/test.gz.gz". + 2. This doesn't extend very well to PUT, because there the order should + basically be the other way around, but the "input" to the job as a whole + should go to the "gzip:/" job, not to the "file:/tmp/test.gz.gz." + It would probably be easier to implement such a job in the way the + current "CopyJob" is done. Have a Job and make all sub-urls sub-jobs of + this Job. + 3. As a result of 1. COPY FROM an url like file:/tmp/test.gz#gzip:/ should + work. COPY TO does not, because that would require PUT. + + +Resuming +-------- + +A rough note for now, just to have this somewhere : +(PJ=put-job, GJ=get-job) + +PJ can't resume: +PJ-->app: canResume(0) (emitted by dataReq) +GJ-->app: data() +PJ-->app: dataReq() +app->PJ: data() + +PJ can resume but GJ can't resume: +PJ-->app: canResume(xx) +app->GJ: start job with "resume=xxx" metadata. +GJ-->app: data() +PJ-->app: dataReq() +app->PJ: data() + +PJ can resume and GJ can resume: +PJ-->app: canResume(xx) +app->GJ: start job with "resume=xxx" metadata. +GJ-->app: canResume(xx) +GJ-->app: data() +PJ-->app: dataReq() +app->PJ: canResume(xx) +app->PJ: data() + +So when the slave supports resume for "put" it has to check after the first +dataRequest() whether it has got a canResume() back from the app. If it did +it must resume. Otherwise it must start from 0. + + |