gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6010 commits 28 branches 278 tags 135 MiB
Commit graph

305 commits

Author SHA1 Message Date
Chris Lu 0ca68a2a6d WIP 2020-03-01 22:13:15 -08:00
Chris Lu 555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu 892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu bd3254b53f adjust logging 2020-02-25 17:24:08 -08:00
Chris Lu 0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu 6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu e96f85ed8a make pb files 2020-02-23 18:06:34 -08:00
Chris Lu 346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server 
Update stats and status volume server
 2020-02-23 18:05:25 -08:00
Chris Lu bbb163c5ad add for https://github.com/chrislusf/seaweedfs/pull/1207 2020-02-23 18:04:31 -08:00
LazyDBA247-Anyvision ec8619176a volume server protobuf: add precentFree & percentUsage to the DiskStatus 2020-02-23 23:24:24 +02:00
Chris Lu f55f49970f move volume server stats to grpc 2020-02-21 21:45:03 -08:00
Chris Lu 0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl 
Add s3 ACL
 2020-02-17 12:53:30 -08:00
Chris Lu 4cdde5f569 configuration stores the identity list 2020-02-17 12:31:59 -08:00
Chris Lu 9b6296e77a volume: add grpc file read operation 
This is added more for performance benchmarking.
 2020-02-14 00:37:32 -08:00
Chris Lu 9ed364f053 support acl 2020-02-09 14:30:02 -08:00
Chris Lu 40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu 03ab0b1b9a correct .vif file creation 2019-12-28 21:40:12 -08:00
Chris Lu 0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu 37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu c06f7eb48a load volume info from .vif file, use superblock as a backup 2019-12-28 12:28:58 -08:00
Chris Lu 2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu b3c83f9404 fix spelling 2019-12-28 11:16:10 -08:00
Chris Lu 48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
Chris Lu d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
Chris Lu 09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu 0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu 1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu 70648d35ad go fmt 2019-12-02 20:49:58 -08:00
Chris Lu 126fb492d2 fix test 2019-12-02 15:52:33 -08:00
Chris Lu ec8de250e2 tiered storage: can copy to s3, read from s3 
master not aware tiered volume yet, file assigning is not working yet
 2019-12-02 15:08:31 -08:00
Chris Lu 228231f3d7 minor 2019-12-02 15:08:31 -08:00
Chris Lu 0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu 586798ecc0 go fmt 2019-11-27 12:34:57 -08:00
Chris Lu 0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Chris Lu 471902cd51 change parameter name 2019-11-10 12:05:45 -08:00
zhangsong 61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00
Chris Lu faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
joeslay d53aee179b
Merge pull request #10 from chrislusf/master 
merge seaweed master
 2019-10-14 16:03:40 +01:00
Chris Lu f8d4b7d1c0 support basic json filtering and selection 2019-10-06 22:35:08 -07:00
Chris Lu cf47f657af scaffold for volume server query feature 2019-10-06 22:35:08 -07:00
j.laycock 6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
joeslay d8c34b032f
Merge pull request #6 from chrislusf/master 
merge seaweed master
 2019-09-12 11:44:09 +01:00
Chris Lu ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
j.laycock 84f2dc3b53 Remove test code 2019-09-05 11:14:02 +01:00
Tom Maxwell 4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock 9aa0859697 Remove inmemory from vaccum stuff 2019-09-03 18:20:45 +01:00
j.laycock cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
Tom Maxwell d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
j.laycock 595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
Chris Lu 19360ab10a master: redirect clients to the new leader 2019-07-31 01:54:42 -07:00
Chris Lu 6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu 8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu 1babec00e7 check deleted chunks faster 2019-06-22 13:22:22 -07:00
Chris Lu cd45ab072a fix compilation error 2019-06-22 12:30:08 -07:00
Chris Lu a111f26fe6 avoid nil 
fix https://github.com/chrislusf/seaweedfs/issues/988
 2019-06-21 20:56:27 -07:00
Chris Lu 059ef879a8 fix issue 986 
fix issue 986
 2019-06-21 13:06:04 -07:00
Chris Lu f88a8bda7b ec deletion works 2019-06-21 01:14:10 -07:00
Chris Lu 4cea8aefd0 add grpc VolumeEcBlobDelete 2019-06-20 00:17:11 -07:00
Chris Lu d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu 450f4733ce report empty volume or ec shards 2019-06-05 13:32:33 -07:00
Chris Lu 7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu f919d0235c ec encode volumes quiet for a period of time 2019-05-31 23:41:17 -07:00
Chris Lu 1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu 3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu 2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu b4b407e403 add grpc ec shard read 2019-05-27 11:59:03 -07:00
Chris Lu db94a41f9e mount/unmount ec shards 2019-05-25 23:23:19 -07:00
Chris Lu 41e8ae61f8 generate, balance, delete copied shards, delete old volume 2019-05-25 14:02:06 -07:00
Chris Lu 6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu 7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Chris Lu 17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu fbbc74abb4 adds VolumeEcGenerateSlices, VolumeEcCopy 2019-05-20 00:53:17 -07:00
Chris Lu 698dea779c refactoring 2019-05-19 21:37:49 -07:00
Chris Lu bf9d490538 fix for tests 2019-05-17 02:28:20 -07:00
Chris Lu 82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu 3ff92037cc shell: add volume.balance command 2019-05-05 21:17:23 -07:00
Chris Lu 316bd27f75 add remove volumes with version info 2019-04-20 23:53:37 -07:00
Chris Lu 2ffe98443b refactoring 2019-04-20 12:05:28 -07:00
Chris Lu 6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
Chris Lu 3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu fa176fe80f volume tailing chunks large files 2019-04-18 19:22:13 -07:00
Chris Lu 3dce1016cb add volume tailer 2019-04-18 11:05:02 -07:00
Chris Lu b09e8dbb37 add VolumeStreamFollow, but not used yet 2019-04-18 00:18:29 -07:00
Chris Lu 13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu cb2061f86a
Merge pull request #925 from stlpmo-jn/checkUnhealthReplication 
the implement of repair the error replications of the volume
 2019-04-10 23:37:12 -07:00
Chris Lu 2f76681d62 refactor: adjust grpc API name 2019-04-10 21:41:17 -07:00
stlpmo-jn c1a0403da2 repair the error replications of the volume 2019-04-10 19:41:55 +08:00
Chris Lu 97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu 19728fe3f6 remove deprecated code 2019-03-25 23:12:14 -07:00
Chris Lu 70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu 95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu ece9d13312 volume info collect compact revision number 2019-03-18 09:32:21 -07:00
Chris Lu aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu 657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu 74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu 4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Chris Lu 0c932d1738 add volume delete grpc API 
fix https://github.com/chrislusf/seaweedfs/issues/808
 2018-12-29 00:03:30 -08:00
Chris Lu 253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu 5333f2984a streaming updates for large entries or large index file 
fix https://github.com/chrislusf/seaweedfs/issues/801
 2018-12-22 11:10:08 -08:00
Chris Lu 39ea8484c1 adjust proto API 2018-12-11 07:17:10 -08:00
Chris Lu 1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu 444dfded84 add fs.FSStatfser for SeaweedFS weed mount 2018-11-23 00:24:51 -08:00
Chris Lu 5065d4ab2d master add grpc API for fileid assigning 2018-11-20 11:35:45 -08:00
Chris Lu 46eb77f9bb move DiskStatus and MemStatus to protobuf 2018-10-15 22:25:28 -07:00
Chris Lu eec951cad2 migrate volume sync to gRpc 2018-10-15 21:44:41 -07:00
Chris Lu f8b2d3cacc move volume mount/unmount on volume server to grpc 2018-10-15 01:48:15 -07:00
Chris Lu fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu 8301519fb0 migrate delete collection to grpc API on volume server 2018-10-15 00:03:55 -07:00
Chris Lu b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu 556382ff5f adding basic grpc to volume server 2018-10-11 01:16:33 -07:00
Chris Lu 64ed47502b rename file 2018-10-11 00:21:32 -07:00
Chris Lu 60c1ada4c1 remove filer grpc API GetEntryAttributes 2018-09-25 09:18:06 -07:00
Chris Lu 788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu 164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu dd37f0540f add extended attributes 2018-09-07 13:12:52 -07:00
Chris Lu ce6a2140a2 preparing to support S3 multipart uploads 2018-09-04 00:42:44 -07:00
Chris Lu 76cbe8bf33 instant notification of new volumes added or deleted 2018-08-24 01:26:56 -07:00
Chris Lu f036ef8a3c add filer notification 2018-08-13 01:20:49 -07:00
Chris Lu a12c7b86b0 broadcast messages of new and deleted volumes 2018-07-27 23:09:55 -07:00
Chris Lu 0d98949199 tmp commit 2018-07-27 01:54:45 -07:00
Chris Lu 6319d84f42 s3 API add ListObjectsV1 2018-07-22 01:14:36 -07:00
Chris Lu 568132c0c9 delete bucket also removes collection data 2018-07-20 00:10:01 -07:00
Chris Lu c7228fefa7 add bucket creation and deletion 
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
 2018-07-19 01:21:44 -07:00
Chris Lu 842dab07b4 weed mount can request to prioritize to write to a data center 2018-07-14 13:36:28 -07:00
Chris Lu 28e5f20c8e add super block extra for erasure coding 2018-06-28 22:20:37 -07:00
Chris Lu f339e9c284 add support for extra super block info 2018-06-24 15:19:57 -07:00
Chris Lu 5bd72696ac weed mount add ttl option 2018-06-11 23:13:33 -07:00
Chris Lu 98110c1697 add collection, replication, ttl to each file 2018-06-10 16:57:32 -07:00
Chris Lu 5c25d29272 support renaming files 2018-06-06 22:11:01 -07:00
Chris Lu 43e3f5724c use fixed list of masters in both filer and volume servers 2018-06-01 00:39:39 -07:00
Chris Lu 0301504184 add mime, use simple insert and update filer store API 
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
 2018-05-30 20:24:57 -07:00
Chris Lu 6de84c64c6 adding create time 2018-05-25 23:26:40 -07:00
Chris Lu d773e11c7a file handler directly read from volume servers 
this mostly works fine now!

next: need to cache files to local disk
 2018-05-24 01:22:37 -07:00
Chris Lu 873868cc10 not working now 
need to add file handler
 2018-05-22 04:31:44 -07:00
Chris Lu 7362de9a18 weed mount can work well 
TODO: somehow filer url is returning empty content
 2018-05-22 03:26:38 -07:00
Chris Lu 9dd228747c filer copy added uid/gid 2018-05-21 01:25:30 -07:00
Chris Lu f07482382b able to update file content 
having some issue when vi reports file changed.
 2018-05-21 00:00:28 -07:00
Chris Lu e31c514b00 adding modified time to file chunk 2018-05-16 00:54:44 -07:00
Chris Lu b303a02461 cp file can work 
1. consolidate to filer_pb.FileChunk
2. dir add file, mkdir
3. file flush, write

updates having issue
 2018-05-16 00:08:44 -07:00
Chris Lu 9f345da20f mv filer proto to filer_pb 2018-05-09 23:18:02 -07:00
Chris Lu 942c2cbd7b mv pb to master_pb 2018-05-09 23:11:54 -07:00
Chris Lu 43a69d20bf change filer API to gRPC 2018-05-08 01:59:43 -07:00
Chris Lu 5b0f5f456e volume server get notified of leader change 2017-01-18 09:34:27 -08:00
Chris Lu adc8d344a5 update proto buffer 2017-01-10 23:57:16 -08:00
Chris Lu e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00