How to upload or download very large files with out any issue ?

How to upload or download very large files with out any issue ?

Things to consider for uploading and downloading the large files to UCM

Many configuration required for dealing with very large file ( more than 10gb) . It also involves many factors like network speed , UCM server capacity , weblogic configuration etc . This blog to help get more data and set the configuration

1. Make server Ready

12C server is ready by default . No need to install any patches on weblogic server and on Oracle_ECM

For 11g server , Below patches should be installed

For upload : 16083651 & 16960063

For Download : 14339868 ( this is only for Native File download , web layout works with out patch)

2. Setting required in the client

In 64 bit machine

Use Chrome 64 bit browser , that is best option for uploading file larger than 4 GB

IE 11 , Latest FireFox might work

In 32 bit machine

It allows only 2 GB size for upload . So we have to enable Chunking for upload. It is basically split the files based on the value set . Server will club all the parts together . It is more time consuming than browser upload .

It is based on Java applet . So browser and your network should support applet usage

To enable Chunking , add below entries in config.cfg of UCM

AppletChunkThreshold=10000000 // size in bytes , chucking will start only after this size

AppletChunkSize=10000000 // each size of the packets send from client

ChunkedRequestTrace=true

MultiUpload=true

DisableHttpUploadChunking=false

3. Setting Max time for weblogic thread

We have to increase the thread time out in weblogic console . By default value is 10 min ( 600 sec), if the upload or download take more than 10 min , weblogic will kill the thread ,upload process will stopped .

How to set : Weblogic console → Servers → UCM server → Configration → Tunning

Set : Stuck Thread max time

4. How to get Value for “Stuck Thread max time “

This is specific to each environment . Get the maximum file to upload or download

Test this RIDC upload code .

Get the RIDC code mention here :

http://wccandlinux.blogspot.in/2016/04/ridc-sample-code-toget-network-speed.html

it will show the network speed and Server Capacity , So “ Stuck Thread max time” should be more than total time required for upload . In the example

Time took in UCM server is 45.422 sec

Processing Capacity of your environment 34459.44696402624 KB/s

Processing Capacity of your environment 34.45944696402624 MB/s

So here no need to change the value . If it more than 600 sec , then “ Stuck Thread max time” should be edited .

5. Get the optimal value for AppletChunkThreshold & AppletChunkSize

We have to do trail and error method . Providing details from different test cases

Size of the file to upload to UCM : 13 gb

Upload Method : Using Chunk Applet

Test Case 1 :
AppletChunkThreshold=10000000
AppletChunkSize=10000000  ( 10 MB)
StuckThreadMaxTime=600 sec ( 10 min) (default value )
Even thoug it shows Checkin is fine but no time spend in the logs . Which means check in is failed . And it failed due to weblogic thread time out .
Test Case 2.a :
AppletChunkThreshold=10000000
AppletChunkSize=1000000000  ( 1GB)
Upload failed in the client system itself

Test Case 2.b :
AppletChunkThreshold=10000000
AppletChunkSize=500000000  ( 500 MB)
Upload cause performance issues in in client . So I stopped the upload

Test Case 3 :
AppletChunkThreshold=10000000
AppletChunkSize=100000000  ( 100 MB)
This causing performance issues in server . Server CPU went up to 100%

Test Case 4 :
AppletChunkThreshold=10000000
AppletChunkSize=50000000  ( 50 MB)
StuckThreadMaxTime=900 sec ( 15 min)
Checkin failed in the system due to thread time out in weblogic .
Last CHUNKED_UPLOAD finished it . It took 528 sec
CHUNKED_UPLOAD [dUser=weblogic][IsJava=1] 528.179931640625(secs)
Test Case 5 :

AppletChunkThreshold=1000000000
AppletChunkSize=40000000 ( 40 MB)
StuckThreadMaxTime=1200 sec ( 20 min)
I am able to check in the file in the UCM with this settings .
Total time : 1070.6832275390625(secs)
Last Chunk : 526.7379150390625(secs)
>requestaudit/6 03.11 11:54:51.750 IdcServer-10 CHECKIN_NEW [dID=1054][dDocName=STJACOBPC1IDCO001218][dDocTitle=WSS][dUser=weblogic][dSecurityGroup=Public][StatusCode=0][StatusMessage=Successfully checked in content item 'STJACOBPC1IDCO001218'.][IsJava=1] 1070.6832275390625(secs)

>requestaudit/6 03.11 11:54:51.763 IdcServer-349 CHUNKED_UPLOAD [dUser=weblogic][IsJava=1] 526.7379150390625(secs)

Last Chunk is the process which will club all the parts together , it is really time consuming process .This can be saved if 64 bit chrome browser is used 

RIDC sample code to get Network Speed , UCM server capacity ,upload status

RIDC sample code to get Network Speed , UCM server capacity ,upload status 

This is mainly intended for large file upload ( 1.5 GB+).

JDK 1.8+ required .

Purpose : Sample code to provide

1.Status of upload process

2.Speed of network

3. Capacity of UCM server

UCM checkin process has 3 main file process

1. Send file to UCM server , UCM server keeps this file in temp directory of UCM server

2. Copy this temp file to Native File Location

3. Delete the Temp file created

4. Wait for the conversion and if there is no conversion copy Native File to Web layout

To get valid response from UCM server step 1,2,3 should be over . This code will help us to get speed of step1 ( Network speed ) , step 2&3 for the UCM capacity

Detailed Steps :

1. Download code  from : https://github.com/sterin501/CheckinSpeed.git

2.Edit config.properties

with URL , primary file etc

3. Run the code like

. ./classpath

java SpeedTest

4. Sample output ( for 1.5 GB file )

Size of file in bytes 1602782395

Size of file in KB 1565217.0

Size of file in MB 1528.5322265625

Starting Uploadler

Sending File to UCM ----->

hostname is 10.184.36.144

Sec is myrealm

Time out value is 780000

@Properties LocalData

UserDateFormat=iso8601

IdcService=CHECKIN_UNIVERSAL

UserTimeZone=UTC

dDocTitle=Test RIDC Checkin 1565217.0

dDocType=Document

dDocAccount=

dSecurityGroup=Public

@end

----->5% speed 78260.85 KB/sec ETA 19.0 sec

----->14% speed 109565.19 KB/sec ETA 12.285714285714285 sec

----->23% speed 119999.97 KB/sec ETA 10.043478260869565 sec

----->32% speed 125217.36 KB/sec ETA 8.5 sec

----->42% speed 131478.228 KB/sec ETA 6.904761904761905 sec

----->50% speed 130434.75 KB/sec ETA 6.0 sec

----->60% speed 134161.45714285714 KB/sec ETA 4.666666666666668 sec

----->68% speed 133043.445 KB/sec ETA 3.7647058823529402 sec

----->68% speed 118260.84 KB/sec ETA 4.23529411764706 sec

----->75% speed 117391.275 KB/sec ETA 3.333333333333334 sec

----->76% speed 108142.26545454546 KB/sec ETA 3.473684210526315 sec

----->76% speed 99130.41 KB/sec ETA 3.7894736842105257 sec

----->77% speed 92709.00692307693 KB/sec ETA 3.8831168831168803 sec

----->77% speed 86086.935 KB/sec ETA 4.181818181818183 sec

----->77% speed 80347.806 KB/sec ETA 4.480519480519483 sec

----->77% speed 75326.068125 KB/sec ETA 4.779220779220779 sec

----->77% speed 70895.12294117647 KB/sec ETA 5.077922077922079 sec

----->77% speed 66956.505 KB/sec ETA 5.376623376623375 sec

----->77% speed 63432.478421052634 KB/sec ETA 5.675324675324674 sec

----->77% speed 60260.8545 KB/sec ETA 5.974025974025974 sec

----->77% speed 57391.29 KB/sec ETA 6.272727272727273 sec

----->79% speed 56205.51954545455 KB/sec ETA 5.848101265822784 sec

----->92% speed 62608.68 KB/sec ETA 2.0 sec

-->100%

time took only for file transfer is 23 sec

Network Speed between client & UCM server is 68052.91304347826 KB/sec

Network Speed between client & UCM server is 66.45792289402173MB/sec

ContentID is STJACOBPC1IDCO001819

Time took in UCM server is 45.422 sec

Processing Capacity of your environment 34459.44696402624 KB/s

Processing Capacity of your environment 34.45944696402624 MB/s

Java details :

1. Java main class to define the thread class

2. In the thread demo class , start () should contain the upload process

3. run () used to track the upload process

4. There will be different class which deals  with RIDC action should be refer by step 2 

How to get Folder Structure including File details from WCC server using RIDC

How to get Folder Structure including File details from WCC server using RIDC 


purpose : To get a folder strucutre from FF enabled WCC server

Output will show Folder details with Parent Folder and contentID belongs to each folder 


RIDC java class :  https://github.com/sterin501/FolderStructure

1. Edit the config.properties file
2. Run the code like java FldStructure FLD_ROOT

IdcService used : FLD_BROWSE

Example : output

java FldStructure FLD_ROOT --> Shows Parent and child folder && <--> Shows the contents



FLD_ROOT-->C9394B7D72DB6648F6DE0ACB220E6258<---> C9394B7D72DB6648F6DE0ACB220E6258-->DB9C18541DDA74BEBFBEE5F008D2FB03<--->PS7003005,PS7003001,PS7003010,PS7003011,PS7003009,PS7003007,PS7003003,PS7003002,PS7003008, FLD_ROOT-->BE91ED870AA7F1DECC857E3FCEDA0C5D<---> BE91ED870AA7F1DECC857E3FCEDA0C5D-->B4730B5CDCD9217F9D7EC85DDD464303<---> FLD_ROOT-->FLD_ENTERPRISE_LIBRARY<---> FLD_ENTERPRISE_LIBRARY-->BDF314BBFB0735189BA579633B8F52F1<---> BDF314BBFB0735189BA579633B8F52F1-->139563A86D0A8557B40228E687AC841D<---> FLD_ENTERPRISE_LIBRARY-->06767B69F8092A70DFA370E65C8D5991<--->PS7008822, FLD_ENTERPRISE_LIBRARY-->24F3ADC920D186FA683D16837133F225<---> 24F3ADC920D186FA683D16837133F225-->61AADBD3C3341654311E113EFCC1055F<---> FLD_ENTERPRISE_LIBRARY-->E9D9BF42F08586C8FEA763DFC45E9492<--->PS7008222, E9D9BF42F08586C8FEA763DFC45E9492-->79B3EBBC303C17D1F66EC0CA9D7F331B<---> ........................................ .......................................

Total Folders Under FLD_ROOT : 29 Total Contents Under FLD_ROOT : 59 Folder Mapping FLD_ROOT-->C9394B7D72DB6648F6DE0ACB220E6258-->DB9C18541DDA74BEBFBEE5F008D2FB03 FLD_ROOT-->BE91ED870AA7F1DECC857E3FCEDA0C5D-->B4730B5CDCD9217F9D7EC85DDD464303 FLD_ROOT-->FLD_ENTERPRISE_LIBRARY-->BDF314BBFB0735189BA579633B8F52F1-->139563A86D0A8557B40228E687AC841D FLD_ROOT-->FLD_ENTERPRISE_LIBRARY-->06767B69F8092A70DFA370E65C8D5991

:Second

java FldStructure BDF314BBFB0735189BA579633B8F52F1 --> Shows Parent and child folder && <--> Shows the contents BDF314BBFB0735189BA579633B8F52F1-->139563A86D0A8557B40228E687AC841D<---> ...................... Total Folders Under BDF314BBFB0735189BA579633B8F52F1 : 1 Total Contents Under BDF314BBFB0735189BA579633B8F52F1 : 1 Folder Mapping BDF314BBFB0735189BA579633B8F52F1-->139563A86D0A8557B40228E687AC841D

How to convert single managed server to WCC cluster on the same physical machine ?

How to convert single managed server to WCC cluster on the same physical machine ?

How to create Vertical  cluster for WCC server ?

Can we run different nodes of same cluster of WCC in same physical machine ?

By defaultVertical  clustering is not possible for WCC cluster . Intradoc port will be read from config.cfg of cluster and it is same for all the nodes . Because of this two different nodes are not possible to run on same machine .

With the help of customization intradoc port can be assigned based on the same name . So with out any shared file system and different machine , WCC can be scaled up . This steps is almost equal to create new Managed server as UCM_server2 in the same domain and in the same physical machine

P.S : This method  is NOT supported by Oracle

Steps :

1.Install Custome component (filter) to change the intradoc port during statrt up

2. Add new cluster in weblogic and add new server

3. Restart the servers

Detailed steps ( Here converting single node to WCC cluster )

1. Install Custome component (filter) to change the intradoc port during statrt up

http://wccandlinux.blogspot.com/2016/03/how-to-change-intradoc-port-ucm-server.html

2. Restart the UCM_server1

3. Access the admin server of the domain

4. Add new cluster as UCM_Cluster ( use default options)

                 



















 5. Create new server as UCM_server2

6. Add the UCM_server2 to UCM_cluster

7. Point the CSDS data source to cluster

8. Change targets of below applications from UCM_server to UCM_Cluster

1. Oracle Universal Content Management - Content Server

2. Oracle UCM Web Services

3. Oracle UCM Native Web Services

4. DMS applications

5. Oracle UCM help

6.wsil-ws

9. Restart the admin server

10.Set the new intradocport for UCM_server2 in config.cfg as

UCM_Server2IntradocServerPort

11. start the UCM_server2 like

./startManagedWebLogic.sh UCM_server2 t3://adminserver:port

12. Verify the intradoc port UCM_Server2

Either by

a. netstat -anp | grep <port>

or

b. Configuration info page

13. Start the UCM_server1

Make sure that only one server is restarted at a time . If customer want disable indexer in one node then further customazation is required like step 1