View Issue Details

IDProjectCategoryView StatusLast Update
0006320MediaMonkey (current)Framework: Scripts/Extensionspublic2010-12-11 00:05
Reporterpeke 
PriorityhighSeverityfeatureReproducibilityN/A
Status closedResolutionfixed 
Product Version 
Target Version4.0Fixed in Version4.0 
Summary0006320: Framework: Retrieve URL data
DescriptionIn MM4 we have added new download manager and MediaMonkey identify itself as an app that access internet and retrieve data.

Currently there is no way to retrieve data from URL using scripting. Adding this will enhance MediaMonkey functionality and enable Script developers to use MM UI for creating custom Panels and Dialogs based on information retrieved from url.

By adding few Scripting functions we will create possibility, so that Script developers can create server side applications and retrieve custom data from their sites or parse HTML pages directly in MM for viewing.
TagsRevision
Fixed in build1307

Relationships

related to 0004976 closedLudek Extensions: Support for Additional browser header info 
related to 0003360 closedLudek Use single internet access library instead of TIE + Indy 

Activities

peke

2010-04-03 15:51

developer   ~0020081

Last edited: 2010-04-03 20:02

To make things future proof I suggest to add new methodfunction CreateWebSystem under SDBTools:
Function CreateWebSystem As SDBWebSystem

peke

2010-04-03 19:56

developer   ~0020083

Last edited: 2010-04-03 20:03

CoClass SDBWebSystem/ISDBWebSystem members would be:
AssignURL = Property Get/Let AssignURL As String
ExecuteURL = Method that Creates background thread to retrieve Data From URL
RawData = Property Get RawData As String (Raw stream returned by ExecuteURL)
SizeData = Property Get SizeData as Integer (Total Number of Bytes if known)
LogInfo = Property Get LogInfo as Integer (HTML error code returned by last ExecuteURL Default 0 or -1)
Timeout = Property Get/Let Timeout As String (in Seconds default 30)

Events:
OnProgress(WebSystem As SDBWebSystem) = Event that will fire on each downloaded part (not sure how indy handle that but in TIE is aprox 4KB) this can be used for creating progress/status bars
OnError(WebSystem As SDBWebSystem) = Triggered in case of completion or in case of any error where WebSystem.LogInfo will contain error code

peke

2010-04-03 20:10

developer   ~0020084

Assigned to Rusty to evaluate priority as my thought is that it should be included in first MM 4.0 public release and hammer any possible issues.

NOTE: this will prepare MediaMonkey for upcomming addons site and ability to show addon categories within MM tree along with browsing addons thru MediaMonkey

peke

2010-04-03 20:12

developer   ~0020085

Reminder sent to: jiri, Ludek, rusty

This is sent due the fact that development time needs to be evaluated.

Ludek

2010-04-12 10:08

developer   ~0020097

I think that something like
DownloadFile( URL as String, FileName as String, ShowProgress as Boolean)
would be enough.

i.e. scripter could use the function to download data to a custom file and then read it from the file.

peke

2010-04-15 22:10

developer   ~0020106

You are right that would be enough for now.

I only wonder if that could be made in separate thread so that downloading would not lock MM until file is finished.

I would suggest that we add event that will be fired apron file download finishes.

example could lock MM further script execution:
'-------
DownloadFile("http://download.microsoft.com/download/1/E/5/1E5F1C0A-0D5B-426A-A603-1798B951DDAE/VS2010Express1.iso", "d:\VS2010Express1.iso",True)
'this line will not be executed until DL is done and
SDB.ProcessMessages 'would not be executed to allow other background script to work which will eventually lead to MM Freeze
'-------

My approach would would allow something like this:
'-------
Dim DLfile
Set DLfile = SDB.Tools.WebSystem.CreateNew
Script.RegisterEvent DLfile, "OnError", "ErrorHandler"
DLfile.AssignURL = "http://download.microsoft.com/download/1/E/5/1E5F1C0A-0D5B-426A-A603-1798B951DDAE/VS2010Express1.iso"
DLfile.ExecuteURL
 
Sub ErrorHandler(obj)
'Handling of error/completion of DL
'...
Script.UnRegisterEvents DLfile
End Sub
'-------

Let me know what you think.

Ludek

2010-05-20 21:48

developer   ~0020173

Last edited: 2010-05-20 21:54

I thought that main purposes for scripters should be to have access to the integrated downloader (that is new in MM 4.0). That way I would add new object ISDBDownloader and several new methods like:

Dim Downloader as ISDBDownloader
Downloader = ISDBApplication::Downloader

Downloader.DownloadFile( URL as String, FileName as String, AddToLibrary as Boolean)

Downloader.GetDownloadsCount as Integer
Downloader.GetDownloadProgress( URL as String) as Integer
Downloader.PauseDownload( URL as String)
Downloader.ResumeDownload( URL as String)
Downloader.CancelDownload( URL as String)

That way scripter could easily initiate download of a new audio/video file into library that will be shown under 'Downloads' node in MM UI and can check its progress.

I understand that it could be enhanced for the purposes to download a file that wouldn't be shown under Downloads node or a short file that is supposed to be downloaded immediatelly. For purposes like this we can just add additional 2 input parameters to achive this:

Downloader.DownloadFile( URL as String, FileName as String, AddToLibrary as Boolean, ShowInUI as Boolean, WaitForDownload as Boolean)

if ShowInUI = false then it won't be shown under Downloads node
if WaitForDownload = true then it won't be downloaded on the background


Nevertheless I am not sure how valuable for scripters these new functions would be, probably nothing at a very high priority.

peke

2010-05-23 00:15

developer   ~0020179

Basically what I want to get with this is to allow scripters to make scripts like this http://www.mediamonkey.com/forum/viewtopic.php?f=2&t=47918&start=0 and retrieve info from internet specifically formatted for MM usage.

will bring up small PHP script along with MM VBScript that will show you exactly what teh end result would be. (Note that it will only work MM4.0 and up)

peke

2010-05-26 03:14

developer   ~0020187

Both PHP and VBS is uploaded to FTP.
I also created small App that will show PHP behavior under access from different browsers.

Example is made emulate MediaMonkey accessing URL Using

DownloadFileEX( URL as String, ShowProgress as Boolean) as String;
but
DownloadFile( URL as String, FileName as String, ShowProgress as Boolean);
can be used also.

NOTE: Return Data Can be used to create Custom Menus/Forms/Tree Entries in MM
On other side Web Developers can add simple code that will format Web page differently in case it is accessed by MM and/or limit accessing to online content only within MM and open possibility to make scripts that interacts with some online resource/site/url.

Ludek

2010-05-26 15:22

developer   ~0020189

Last edited: 2010-05-28 12:35

Peke,
Re: the topic you posted ( http://www.mediamonkey.com/forum/viewtopic.php?f=2&t=47918&start=0 ) is long and I don't know what exactly you are referring to. As I know the Lyricator script uses its own internet access to download the lyrics.

Re: the posted example scripts
If I understand correctly you are reffering to that MediaMonkey sets user-agent: MediaMonkey in the HTTP request header and server can format the data based on the info. Similar way could the scripts set the request header or to use the MM API functions like DownloadFile as you pointed.

So do you agree with API I suggested in the note 0020173?
What do you mean by
DownloadFileEX( URL as String, ShowProgress as Boolean) as String;
the function is missing destination filename (where to download the file) so how this should work?
 Should it download the file to the default download location specified in Options?

peke

2010-05-29 23:47

developer   ~0020197

1. Re lyricator: Script was important not the whole post as like you noticed it uses own internet access to download the lyrics which can cause UAC and IE setting issues. Direct result download using MM API should solve those issues.

2. Re example scripts: Yes, as you solved 0004976 the point is where using MM API functions like DownloadFile will tell servers how to respond to HTTP requests both POST and GET and give Scripters ability to make their own versions we talked about in 0003360 NoteID: 18515, 18531 and 18518

For 18518 affiliate sites are directly noted about access from MM due the 0004976 and can make smaller changes if needed to return results back to MM like we talked about in 18515 and XML server results according to common API.

3. I agree with you with proposal in 0020173 but I think it is not currently needed due the right observation of how much of that features is actually needed.

4. re DownloadFileEX: It does not save file at all it returns RAW DATA result received from parsed URL parameter.

REGRESSION WARNING: Due the design nature that both DownloadFileEX and DownloadFile blocks scripts execution until function is done MM should execute http://www.mediamonkey.com/wiki/index.php/ISDBApplication::ProcessMessages so the other scripts and MM threads/background processes are not locked.

Ludek

2010-06-02 13:20

developer   ~0020201

Last edited: 2010-06-02 13:23

Ok, but I would rather call it "GetURLContent" instead of "DownloadFileEX".
And I would add OnDownloadFinished() event that would hit once the download (previously initialized by Downloader.DownloadFile) finishes. So the new API would look like this:


Dim Downloader as ISDBDownloader
Downloader = ISDBApplication::Downloader

Downloader.DownloadFile( URL as String, FileName as String, AddToLibrary as Boolean)
Downloader.OnDownloadFinished( URL as String)
Downloader.GetURLContent( URL as String) as String
Downloader.GetDownloadProgress( URL as String) as Integer
Downloader.PauseDownload( URL as String)
Downloader.ResumeDownload( URL as String)
Downloader.CancelDownload( URL as String)

peke

2010-06-03 23:57

developer   ~0020247

I agree on this as best choice, but to cover all aspects I suggest this change:

Downloader.GetDownloadProgress( URL as String, ResultType as Integer) 'Where Second argument should tell if result is an: 0 = Bytes Downloaded (1234567 Bytes), 1 = Percentage (12 %)

Additionally to let scripter control all using script two more functions would be needed:
Downloader.Status( URL as String) 'Return -1 = Unknown URL, 0 = Finished, 1 = Downloading, 2 = Paused

Downloader.URLSize( URL as String) 'Return Estimated URL Stream Size in Bytes reported/returned by server where -1 = Unknown size

peke

2010-06-03 23:58

developer   ~0020248

Script example:
------
Dim Downloader
Downloader = SDB.Tools.Downloader
'download file example
Downloader.Downloadfile("http://www.happymonkeying.com/beta/mmurltst.php","c:\mmurltest.txt",True)

'download url content
Set URLData = Downloader.GetURLContent("http://www.happymonkeying.com/beta/mmurltst.php")

'Retrieve Progress info for URL
Set URLProgress = Downloader.GetDownloadProgress("http://www.happymonkeying.com/beta/mmurltst.php",1) 'Second argument Should tell if result is an: 0 = Bytes Downloaded (1234567 Bytes) Default, 1 = Percentage (12 %)

'Pause Download of URL
Downloader.PauseDownload("http://www.happymonkeying.com/beta/mmurltst.php")

'Resume URL Download
Downloader.ResumeDownload("http://www.happymonkeying.com/beta/mmurltst.php")

'Cancel URL Download
Downloader.CancelDownload("http://www.happymonkeying.com/beta/mmurltst.php")

'URL Downloader Status
Set URLStatus = Downloader.Status("http://www.happymonkeying.com/beta/mmurltst.php") '-1 = Unknown URL, 0 = Finished, 1 = Downloading, 2 = Paused

'URL Downloader URL Size
Set URLSize = Downloader.URLSize("http://www.happymonkeying.com/beta/mmurltst.php") 'Estimated URL Stream Size reported by server (-1 = Unknown size)

Sub OnDownloadFinished(arg) 'Downloader.OnDownloadFinished( URL as String) Event that must be in same Script
  If arg = "http://www.happymonkeying.com/beta/mmurltst.php" Then
      'somecode
  Else
    'somecode
  End if
End Sub
------

Ludek

2010-06-08 09:32

developer   ~0020253

Implemented in build 1307 and documented:
http://www.mediamonkey.com/wiki/index.php/ISDBApplication::Downloader
http://www.mediamonkey.com/wiki/index.php/ISDBApplicationEvents::OnDownloadFinished

peke

2010-06-08 16:16

developer   ~0020254

Last edited: 2010-06-08 16:16

You are missing Status function.

Without it there is no way to make scripts like this where OnDownloadFinish is not needed and there is no error handling in scripts.

Example:
----
Option Explicit
 
Dim UI : Set UI = SDB.UI
 
Sub OnStartUp()
    Dim mnuTest
 
    Set mnuTest = SDB.UI.AddMenuItem(SDB.UI.Menu_Edit, 0, 0)
    mnuTest.Caption = SDB.Localize("Download Script")
    mnuTest.OnClickFunc = "SDBOnClick"
    mnuTest.UseScript = Script.ScriptPath
 
End Sub

Sub SDBOnClick(Item)
  SDB.Downloader.DownloadFile "http://www.spiker.wz.cz/SecretStream.mp3", "C:\aaa.mp3", False
  Do
    SDB.ProcessMessages
    Sleep(500)
  Loop While SDB.Downloader.Status("http://www.spiker.wz.cz/SecretStream.mp3") = 1
  SDB.ProcessMessages
End Sub
----

Ludek

2010-06-10 13:44

developer   ~0020257

Last edited: 2010-06-10 13:45

Re: Error handling: I've added the Success as second parameter, see:
http://www.mediamonkey.com/wiki/index.php/ISDBApplicationEvents::OnDownloadFinished

Re: Status function: Use GetDownloadProgress(), it returns -1 if the download isn't in progress, see:
http://www.mediamonkey.com/wiki/index.php/ISDBDownloader::GetDownloadProgress

Fixed in build 1307.

peke

2010-06-10 17:45

developer   ~0020258

Last edited: 2010-06-10 17:51

OK what about:
1. IDLE/PAUSED and CANCELED state?
2. No handling of ERROR CODE like 404 Not Found, 403 Denied, ...?
3. No way to get Total length/size of File?

Ludek

2010-06-10 20:17

developer   ~0020259

Last edited: 2010-06-10 20:20

I don't see them much useful. This is just simple API to access integrated download manager. If someone needs an additional functions like custom header sending or detailed error handling then he should use an internet access API or library like Lyricator script does.

peke

2010-06-11 00:08

developer   ~0020260

Last edited: 2010-06-11 00:15

1. In case that there is no any status feedback scripter would need to guess downloader URL status and best solution would be remove separate PauseDownload, ResumeDownload, CancelDownload and add unified method thru Property Get/Let that will look like this adn will allow future expanding:
GetURLStatus Property Get/Let Objects(URL As String) As Integer

GET Results would be:
-1 = URL Unknown
0 = Canceled/Failed
1 = Downloading/Active
2 = Paused/Idle
3 = Query for Resuming
--- Future Use
4 = Waiting for processing (URL present in downloader queue)
5 = In process for removal from downloader queue
? = For future expansions

LET Results would be:
0 = Cancel Download
1 = Download/Retry
2 = Pause
3 = Resume
--- Future Use
4 = Move to end of downloader queue
5 = Remove from downloader queue (in case URL is Downloading/Active Cancel Download before removal)
? = For future expansions

This way we will make things more script developer friendlier.


2. Regarding error handling I agree that we should keep it as basic as possible http://www.mediamonkey.com/wiki/index.php/ISDBApplicationEvents::OnDownloadFinished internet API is not multi platform and request advanced knowledge. Bringing internal passthru of downloader thru Success Parameter would error code (4xx Client errors along with 5xx Server errors for start) based on http://en.wikipedia.org/wiki/List_of_HTTP_status_codes, http://www.iana.org/assignments/http-status-codes and http://www.askapache.com/htaccess/apache-status-code-headers-errordocument.html along with few internal return codes -1 = Download Complete, 0 = Download Canceled. Returning these Codes would evade locking MM in Loop functions and allow Scripter to handle online content more efficiently along with ways to optimize Script code to use much less Resources (CPU) and improve overall MM speed considerably (Small scripts can be made to show resource usage).

HINT: Firefox Have addon for easy live watching/capturing Headers and Responses see https://addons.mozilla.org/en-US/firefox/addon/3829/

NOTE: If it is too complicated for immediate implementation just change Success parameter to Integer instead of Boolean and give return values -1 = Download Complete, 0 = Download Canceled which will allow us later extending without compatibility issues due the fact that TRUE/FALSE represents -1/0 according to http://www.w3schools.com/vbScript/vbscript_ref_keywords.asp and proven true by this Script:
----
Dim test
test = -1
If test = TRUE Then
  test = 0
end If

If test = FALSE Then
  test = -1
End If
----

3. I missed this but Length can be retrieved using http://www.mediamonkey.com/wiki/index.php/ISDBDownloader::GetURLContentLength

4. For Consistence with other Methods in Downloader http://www.mediamonkey.com/wiki/index.php/ISDBDownloader::GetDownloadProgress would be more clear if renamed to GetURLProgress ( http://www.mediamonkey.com/wiki/index.php/ISDBDownloader::GetURLProgress )

Ludek

2010-06-11 08:02

developer   ~0020261

Last edited: 2010-06-14 13:48

1. We can add the GetDownloadStatus() in a future version, but I believe that for now the GetDownloadProgress() is enough. It tells whether the download is in progress and what the progress is.

2. Ok, I extended the event and added the ResponseCode parameter, see http://www.mediamonkey.com/wiki/index.php/ISDBApplicationEvents::OnDownloadFinished

4. I think that GetDownloadProgress() is more intuitive than GetURLProgress(), but this is not a big deal.

peke

2010-12-10 21:05

developer   ~0021724

Verified 1333

Added Revision Tag for future easier locating

peke

2010-12-11 00:04

developer   ~0021726

Last edited: 2010-12-11 00:05

View 2 revisions

I corrected http://www.mediamonkey.com/wiki/index.php/SDBDownloader to have reference to http://www.mediamonkey.com/wiki/index.php/ISDBApplicationEvents::OnDownloadFinished Event for easier implementation and script optimization